You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Wangda Tan <wh...@gmail.com> on 2018/03/30 04:15:12 UTC
[VOTE] Release Apache Hadoop 3.1.0 (RC1)
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Rohith Sharma K S <ro...@apache.org>.
+1 (binding)
* Downloaded source, built from source with -Dhbase.profile=2.0.
* Installed RM HA cluster integrated with ATSv2. Installed HBase-2.0-beta1
for ATSv2 back end. Scheduler is configured with 2 level queue hierarchy.
* Ran sample jobs such as MR/Distributed shell and verified for
** ATSv2 Timeline Reader REST API's. Validated for data published in ATSv2.
** YARN UI2 accessed for Flow Activity pages. Navigated inside flow
activity page for all other info.
** YARN old UI also verified for all the pages
** RM HA/Restart/work-preserving-restart are tested while running a job.
** NM restart scenarios are verified.
** Application timeout and application priority feature is tested.
Thanks & Regards
Rohith Sharma K S
On 30 March 2018 at 09:45, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Eric Payne <er...@yahoo.com.INVALID>.
+1 (binding)
Thanks Wangda for doing the work to produce this release.
I did the following to test the release:
- Built from source
- Installed on 6-node pseudo cluster
- Interacted with RM CLI and GUI
- Tested streaming jobs
- Tested yarn distributed shell jobs
- Tested Max AM Resource Percent
- Tested simple inter-queue preemption
- Tested priority first intra-queue preemption
- Tested userlimit first intra-queue preemption
Thanks,Eric Payne
===========================================================
On Thursday, March 29, 2018, 11:15:51 PM CDT, Wangda Tan <wh...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Rohith Sharma K S <ro...@apache.org>.
+1 (binding)
* Downloaded source, built from source with -Dhbase.profile=2.0.
* Installed RM HA cluster integrated with ATSv2. Installed HBase-2.0-beta1
for ATSv2 back end. Scheduler is configured with 2 level queue hierarchy.
* Ran sample jobs such as MR/Distributed shell and verified for
** ATSv2 Timeline Reader REST API's. Validated for data published in ATSv2.
** YARN UI2 accessed for Flow Activity pages. Navigated inside flow
activity page for all other info.
** YARN old UI also verified for all the pages
** RM HA/Restart/work-preserving-restart are tested while running a job.
** NM restart scenarios are verified.
** Application timeout and application priority feature is tested.
Thanks & Regards
Rohith Sharma K S
On 30 March 2018 at 09:45, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Yufei Gu <fl...@gmail.com>.
Thanks Wangda for working on this!
+1 (non-binding)
- Downloaded the binary tarball and verified the checksum.
- Started a pseudo cluster inside one docker container
- Run Resource Manager with Fair Scheduler
- Verified distributed shell
- Verified mapreduce pi job
- Sanity checked RM WebUI
Best,
Yufei
On Thu, Mar 29, 2018 at 9:15 PM, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
RE: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Takanobu Asanuma <ta...@yahoo-corp.jp>.
Thanks for your efforts, Wangda. Sorry for participating late.
+1 (non-binding)
- Verified the checksum of the tarball
- Succeeded "mvn clean package -Pdist,native -Dtar -DskipTests"
- Started docker hadoop cluster with 1 master and 5 slaves
- Verified TeraGen/TeraSort
- Verified some erasure coding operations
- Seems the shaded jars (hadoop-client-runtime, hadoop-client-api, hadoop-client-minicluster) have correct sizes in orgapachehadoop-1092.
-Takanobu
-----Original Message-----
From: Wangda Tan [mailto:wheeleast@gmail.com]
Sent: Tuesday, April 03, 2018 1:25 PM
To: Arpit Agarwal <aa...@hortonworks.com>
Cc: Gera Shegalov <ge...@gmail.com>; Sunil G <su...@apache.org>; yarn-dev@hadoop.apache.org; Hdfs-dev <hd...@hadoop.apache.org>; Hadoop Common <co...@hadoop.apache.org>; mapreduce-dev@hadoop.apache.org; Vinod Kumar Vavilapalli <vi...@apache.org>
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo?
https://repository.apache.org/content/repositories/orgapachehadoop-1092
Since the jars inside binary tarballs are correct ( http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll another RC, just update Maven repo should be sufficient.
Best,
Wangda
On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <wh...@gmail.com> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums
> still exist in .mds files and I didn't remove them from .mds file
> because it is generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal
> <aa...@hortonworks.com>
> wrote:
>
>> Thanks for putting together this RC, Wangda.
>>
>> The guidance from Apache is to omit MD5s, specifically:
>> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi Gera,
>>
>> It's my bad, I thought only src/bin tarball is enough.
>>
>> I just uploaded all other things under artifact/ to
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>> Please let me know if you have any other comments.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
>>
>> Thanks, Wangda!
>>
>> There are many more artifacts in previous votes, e.g., see
>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others
>> the site tarball is missing.
>>
>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
>>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old
>> RM UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1].
>>
>> We've
>>
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/
>>
>> orgapachehadoop-1090/
>>
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable
>> additions include the first class GPU/FPGA support on YARN, Native
>> services,
>>
>> Support
>>
>> rich placement constraints in YARN, S3-related enhancements, allow
>> HDFS block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending
>> on
>>
>> how
>>
>> it goes, get it stabilized and potentially use a 3.1.1 in several
>> weeks
>>
>> as
>>
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>>
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>> (3.1.0) AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status =
>> Resolved ORDER
>>
>> BY
>>
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>>
>>
>>
>>
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Gera Shegalov <ge...@apache.org>.
+1 (non-binding)
- built from source
- tested SparkPi on minicluster (modulo YARN-7747
<https://issues.apache.org/jira/browse/YARN-7747>)
- tested SparkPi on pseudo-distributed cluster
- browsed HDFS doc in the site tarball
On Tue, Apr 3, 2018 at 11:55 AM Hanisha Koneru <hk...@hortonworks.com>
wrote:
> Correction: My vote is NON-BINDING. Sorry for the confusion.
>
>
> Thanks,
> Hanisha
>
>
>
>
>
>
>
>
> On 4/3/18, 11:40 AM, "Hanisha Koneru" <hk...@hortonworks.com> wrote:
>
> >Thanks Wangda for putting up the RC for 3.1.0.
> >
> >+1 (binding).
> >
> >Verified the following:
> >- Built from source
> >- Deployed binary to a 3-node docker cluster
> >- Sanity checks
> > - Basic dfs operations
> > - MapReduce Wordcount & Grep
> >
> >
> >Thanks,
> >Hanisha
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >On 4/3/18, 9:33 AM, "Arpit Agarwal" <aa...@hortonworks.com> wrote:
> >
> >>Thanks Wangda, I see the shaded jars now.
> >>
> >>Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
> >>
> >>+1 (binding), modulo that remaining question.
> >>
> >>* Verified signatures
> >>* Verified checksums for source and binary artefacts
> >>* Sanity checked jars on r.a.o.
> >>* Built from source
> >>* Deployed to 3 node secure cluster with NameNode HA
> >>* Verified HDFS web UIs
> >>* Tried out HDFS shell commands
> >>* Ran sample MapReduce jobs
> >>
> >>Thanks!
> >>
> >>
> >>----------------------------------------------------------------------
> >>From: Wangda Tan <wh...@gmail.com>
> >>Date: Monday, April 2, 2018 at 9:25 PM
> >>To: Arpit Agarwal <aa...@hortonworks.com>
> >>Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> >>Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> >>
> >>As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo?
> https://repository.apache.org/content/repositories/orgapachehadoop-1092
> >>
> >>Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
> >>
> >>Best,
> >>Wangda
> >>
> >>
> >>On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> >>Hi Arpit,
> >>
> >>Thanks for pointing out this.
> >>
> >>I just removed all .md5 files from artifacts. I found md5 checksums
> still exist in .mds files and I didn't remove them from .mds file because
> it is generated by create-release script and Apache guidance is "should
> not" instead of "must not". Please let me know if you think they need to be
> removed as well.
> >>
> >>- Wangda
> >>
> >>
> >>
> >>On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> >>Thanks for putting together this RC, Wangda.
> >>
> >>The guidance from Apache is to omit MD5s, specifically:
> >> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> >>
> >>https://www.apache.org/dev/release-distribution#sigs-and-sums
> >>
> >>
> >>
> >>
> >>On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
> >>
> >>Hi Gera,
> >>
> >>It's my bad, I thought only src/bin tarball is enough.
> >>
> >>I just uploaded all other things under artifact/ to
> >>http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> >>
> >>Please let me know if you have any other comments.
> >>
> >>Thanks,
> >>Wangda
> >>
> >>
> >>On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
> >>
> >>
> >>Thanks, Wangda!
> >>
> >>There are many more artifacts in previous votes, e.g., see
> >>http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> >>site tarball is missing.
> >>
> >>On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org>
> wrote:
> >>
> >>
> >>Thanks Wangda for initiating the release.
> >>
> >>I tested this RC built from source file.
> >>
> >>
> >> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> >>UI.
> >> - Below feature sanity is done
> >> - Application priority
> >> - Application timeout
> >> - Intra Queue preemption with priority based
> >> - DS based affinity tests to verify placement constraints.
> >> - Tested basic NodeLabel scenarios.
> >> - Added couple of labels to few of nodes and behavior is coming
> >> correct.
> >> - Verified old UI and new YARN UI for labels.
> >> - Submitted apps to labelled cluster and it works fine.
> >> - Also performed few cli commands related to nodelabel.
> >> - Test basic HA cases and seems correct.
> >> - Tested new YARN UI . All pages are getting loaded correctly.
> >>
> >>
> >>- Sunil
> >>
> >>On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
> >>
> >>
> >>Hi folks,
> >>
> >>Thanks to the many who helped with this release since Dec 2017 [1].
> >>We've
> >>
> >>created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >>
> >>http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >>
> >>The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> >>16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >>
> >>The maven artifacts are available via http://repository.apache.org at
> >>https://repository.apache.org/content/repositories/
> >>orgapachehadoop-1090/
> >>
> >>This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >>
> >>3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> >>include the first class GPU/FPGA support on YARN, Native services,
> >>Support
> >>
> >>rich placement constraints in YARN, S3-related enhancements, allow HDFS
> >>block replicas to be provided by an external storage system, etc.
> >>
> >>For 3.1.0 RC0 vote discussion, please see [3].
> >>
> >>We’d like to use this as a starting release for 3.1.x [1], depending on
> >>how
> >>
> >>it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> >>as
> >>
> >>the stable release.
> >>
> >>We have done testing with a pseudo cluster:
> >>- Ran distributed job.
> >>- GPU scheduling/isolation.
> >>- Placement constraints (intra-application anti-affinity) by using
> >>distributed shell.
> >>
> >>My +1 to start.
> >>
> >>Best,
> >>Wangda/Vinod
> >>
> >>[1]
> >>
> >>https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> >>c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
> >>
> >>[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> >>AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> >>BY
> >>
> >>fixVersion ASC
> >>[3]
> >>
> >>https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> >>2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>---------------------------------------------------------------------
> >>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Gera Shegalov <ge...@apache.org>.
+1 (non-binding)
- built from source
- tested SparkPi on minicluster (modulo YARN-7747
<https://issues.apache.org/jira/browse/YARN-7747>)
- tested SparkPi on pseudo-distributed cluster
- browsed HDFS doc in the site tarball
On Tue, Apr 3, 2018 at 11:55 AM Hanisha Koneru <hk...@hortonworks.com>
wrote:
> Correction: My vote is NON-BINDING. Sorry for the confusion.
>
>
> Thanks,
> Hanisha
>
>
>
>
>
>
>
>
> On 4/3/18, 11:40 AM, "Hanisha Koneru" <hk...@hortonworks.com> wrote:
>
> >Thanks Wangda for putting up the RC for 3.1.0.
> >
> >+1 (binding).
> >
> >Verified the following:
> >- Built from source
> >- Deployed binary to a 3-node docker cluster
> >- Sanity checks
> > - Basic dfs operations
> > - MapReduce Wordcount & Grep
> >
> >
> >Thanks,
> >Hanisha
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >On 4/3/18, 9:33 AM, "Arpit Agarwal" <aa...@hortonworks.com> wrote:
> >
> >>Thanks Wangda, I see the shaded jars now.
> >>
> >>Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
> >>
> >>+1 (binding), modulo that remaining question.
> >>
> >>* Verified signatures
> >>* Verified checksums for source and binary artefacts
> >>* Sanity checked jars on r.a.o.
> >>* Built from source
> >>* Deployed to 3 node secure cluster with NameNode HA
> >>* Verified HDFS web UIs
> >>* Tried out HDFS shell commands
> >>* Ran sample MapReduce jobs
> >>
> >>Thanks!
> >>
> >>
> >>----------------------------------------------------------------------
> >>From: Wangda Tan <wh...@gmail.com>
> >>Date: Monday, April 2, 2018 at 9:25 PM
> >>To: Arpit Agarwal <aa...@hortonworks.com>
> >>Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> >>Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> >>
> >>As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo?
> https://repository.apache.org/content/repositories/orgapachehadoop-1092
> >>
> >>Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
> >>
> >>Best,
> >>Wangda
> >>
> >>
> >>On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> >>Hi Arpit,
> >>
> >>Thanks for pointing out this.
> >>
> >>I just removed all .md5 files from artifacts. I found md5 checksums
> still exist in .mds files and I didn't remove them from .mds file because
> it is generated by create-release script and Apache guidance is "should
> not" instead of "must not". Please let me know if you think they need to be
> removed as well.
> >>
> >>- Wangda
> >>
> >>
> >>
> >>On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> >>Thanks for putting together this RC, Wangda.
> >>
> >>The guidance from Apache is to omit MD5s, specifically:
> >> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> >>
> >>https://www.apache.org/dev/release-distribution#sigs-and-sums
> >>
> >>
> >>
> >>
> >>On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
> >>
> >>Hi Gera,
> >>
> >>It's my bad, I thought only src/bin tarball is enough.
> >>
> >>I just uploaded all other things under artifact/ to
> >>http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> >>
> >>Please let me know if you have any other comments.
> >>
> >>Thanks,
> >>Wangda
> >>
> >>
> >>On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
> >>
> >>
> >>Thanks, Wangda!
> >>
> >>There are many more artifacts in previous votes, e.g., see
> >>http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> >>site tarball is missing.
> >>
> >>On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org>
> wrote:
> >>
> >>
> >>Thanks Wangda for initiating the release.
> >>
> >>I tested this RC built from source file.
> >>
> >>
> >> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> >>UI.
> >> - Below feature sanity is done
> >> - Application priority
> >> - Application timeout
> >> - Intra Queue preemption with priority based
> >> - DS based affinity tests to verify placement constraints.
> >> - Tested basic NodeLabel scenarios.
> >> - Added couple of labels to few of nodes and behavior is coming
> >> correct.
> >> - Verified old UI and new YARN UI for labels.
> >> - Submitted apps to labelled cluster and it works fine.
> >> - Also performed few cli commands related to nodelabel.
> >> - Test basic HA cases and seems correct.
> >> - Tested new YARN UI . All pages are getting loaded correctly.
> >>
> >>
> >>- Sunil
> >>
> >>On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
> >>
> >>
> >>Hi folks,
> >>
> >>Thanks to the many who helped with this release since Dec 2017 [1].
> >>We've
> >>
> >>created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >>
> >>http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >>
> >>The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> >>16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >>
> >>The maven artifacts are available via http://repository.apache.org at
> >>https://repository.apache.org/content/repositories/
> >>orgapachehadoop-1090/
> >>
> >>This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >>
> >>3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> >>include the first class GPU/FPGA support on YARN, Native services,
> >>Support
> >>
> >>rich placement constraints in YARN, S3-related enhancements, allow HDFS
> >>block replicas to be provided by an external storage system, etc.
> >>
> >>For 3.1.0 RC0 vote discussion, please see [3].
> >>
> >>We’d like to use this as a starting release for 3.1.x [1], depending on
> >>how
> >>
> >>it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> >>as
> >>
> >>the stable release.
> >>
> >>We have done testing with a pseudo cluster:
> >>- Ran distributed job.
> >>- GPU scheduling/isolation.
> >>- Placement constraints (intra-application anti-affinity) by using
> >>distributed shell.
> >>
> >>My +1 to start.
> >>
> >>Best,
> >>Wangda/Vinod
> >>
> >>[1]
> >>
> >>https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> >>c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
> >>
> >>[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> >>AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> >>BY
> >>
> >>fixVersion ASC
> >>[3]
> >>
> >>https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> >>2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>---------------------------------------------------------------------
> >>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Gera Shegalov <ge...@apache.org>.
+1 (non-binding)
- built from source
- tested SparkPi on minicluster (modulo YARN-7747
<https://issues.apache.org/jira/browse/YARN-7747>)
- tested SparkPi on pseudo-distributed cluster
- browsed HDFS doc in the site tarball
On Tue, Apr 3, 2018 at 11:55 AM Hanisha Koneru <hk...@hortonworks.com>
wrote:
> Correction: My vote is NON-BINDING. Sorry for the confusion.
>
>
> Thanks,
> Hanisha
>
>
>
>
>
>
>
>
> On 4/3/18, 11:40 AM, "Hanisha Koneru" <hk...@hortonworks.com> wrote:
>
> >Thanks Wangda for putting up the RC for 3.1.0.
> >
> >+1 (binding).
> >
> >Verified the following:
> >- Built from source
> >- Deployed binary to a 3-node docker cluster
> >- Sanity checks
> > - Basic dfs operations
> > - MapReduce Wordcount & Grep
> >
> >
> >Thanks,
> >Hanisha
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >On 4/3/18, 9:33 AM, "Arpit Agarwal" <aa...@hortonworks.com> wrote:
> >
> >>Thanks Wangda, I see the shaded jars now.
> >>
> >>Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
> >>
> >>+1 (binding), modulo that remaining question.
> >>
> >>* Verified signatures
> >>* Verified checksums for source and binary artefacts
> >>* Sanity checked jars on r.a.o.
> >>* Built from source
> >>* Deployed to 3 node secure cluster with NameNode HA
> >>* Verified HDFS web UIs
> >>* Tried out HDFS shell commands
> >>* Ran sample MapReduce jobs
> >>
> >>Thanks!
> >>
> >>
> >>----------------------------------------------------------------------
> >>From: Wangda Tan <wh...@gmail.com>
> >>Date: Monday, April 2, 2018 at 9:25 PM
> >>To: Arpit Agarwal <aa...@hortonworks.com>
> >>Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> >>Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> >>
> >>As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo?
> https://repository.apache.org/content/repositories/orgapachehadoop-1092
> >>
> >>Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
> >>
> >>Best,
> >>Wangda
> >>
> >>
> >>On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> >>Hi Arpit,
> >>
> >>Thanks for pointing out this.
> >>
> >>I just removed all .md5 files from artifacts. I found md5 checksums
> still exist in .mds files and I didn't remove them from .mds file because
> it is generated by create-release script and Apache guidance is "should
> not" instead of "must not". Please let me know if you think they need to be
> removed as well.
> >>
> >>- Wangda
> >>
> >>
> >>
> >>On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> >>Thanks for putting together this RC, Wangda.
> >>
> >>The guidance from Apache is to omit MD5s, specifically:
> >> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> >>
> >>https://www.apache.org/dev/release-distribution#sigs-and-sums
> >>
> >>
> >>
> >>
> >>On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
> >>
> >>Hi Gera,
> >>
> >>It's my bad, I thought only src/bin tarball is enough.
> >>
> >>I just uploaded all other things under artifact/ to
> >>http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> >>
> >>Please let me know if you have any other comments.
> >>
> >>Thanks,
> >>Wangda
> >>
> >>
> >>On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
> >>
> >>
> >>Thanks, Wangda!
> >>
> >>There are many more artifacts in previous votes, e.g., see
> >>http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> >>site tarball is missing.
> >>
> >>On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org>
> wrote:
> >>
> >>
> >>Thanks Wangda for initiating the release.
> >>
> >>I tested this RC built from source file.
> >>
> >>
> >> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> >>UI.
> >> - Below feature sanity is done
> >> - Application priority
> >> - Application timeout
> >> - Intra Queue preemption with priority based
> >> - DS based affinity tests to verify placement constraints.
> >> - Tested basic NodeLabel scenarios.
> >> - Added couple of labels to few of nodes and behavior is coming
> >> correct.
> >> - Verified old UI and new YARN UI for labels.
> >> - Submitted apps to labelled cluster and it works fine.
> >> - Also performed few cli commands related to nodelabel.
> >> - Test basic HA cases and seems correct.
> >> - Tested new YARN UI . All pages are getting loaded correctly.
> >>
> >>
> >>- Sunil
> >>
> >>On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
> >>
> >>
> >>Hi folks,
> >>
> >>Thanks to the many who helped with this release since Dec 2017 [1].
> >>We've
> >>
> >>created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >>
> >>http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >>
> >>The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> >>16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >>
> >>The maven artifacts are available via http://repository.apache.org at
> >>https://repository.apache.org/content/repositories/
> >>orgapachehadoop-1090/
> >>
> >>This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >>
> >>3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> >>include the first class GPU/FPGA support on YARN, Native services,
> >>Support
> >>
> >>rich placement constraints in YARN, S3-related enhancements, allow HDFS
> >>block replicas to be provided by an external storage system, etc.
> >>
> >>For 3.1.0 RC0 vote discussion, please see [3].
> >>
> >>We’d like to use this as a starting release for 3.1.x [1], depending on
> >>how
> >>
> >>it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> >>as
> >>
> >>the stable release.
> >>
> >>We have done testing with a pseudo cluster:
> >>- Ran distributed job.
> >>- GPU scheduling/isolation.
> >>- Placement constraints (intra-application anti-affinity) by using
> >>distributed shell.
> >>
> >>My +1 to start.
> >>
> >>Best,
> >>Wangda/Vinod
> >>
> >>[1]
> >>
> >>https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> >>c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
> >>
> >>[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> >>AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> >>BY
> >>
> >>fixVersion ASC
> >>[3]
> >>
> >>https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> >>2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>---------------------------------------------------------------------
> >>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Gera Shegalov <ge...@apache.org>.
+1 (non-binding)
- built from source
- tested SparkPi on minicluster (modulo YARN-7747
<https://issues.apache.org/jira/browse/YARN-7747>)
- tested SparkPi on pseudo-distributed cluster
- browsed HDFS doc in the site tarball
On Tue, Apr 3, 2018 at 11:55 AM Hanisha Koneru <hk...@hortonworks.com>
wrote:
> Correction: My vote is NON-BINDING. Sorry for the confusion.
>
>
> Thanks,
> Hanisha
>
>
>
>
>
>
>
>
> On 4/3/18, 11:40 AM, "Hanisha Koneru" <hk...@hortonworks.com> wrote:
>
> >Thanks Wangda for putting up the RC for 3.1.0.
> >
> >+1 (binding).
> >
> >Verified the following:
> >- Built from source
> >- Deployed binary to a 3-node docker cluster
> >- Sanity checks
> > - Basic dfs operations
> > - MapReduce Wordcount & Grep
> >
> >
> >Thanks,
> >Hanisha
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >On 4/3/18, 9:33 AM, "Arpit Agarwal" <aa...@hortonworks.com> wrote:
> >
> >>Thanks Wangda, I see the shaded jars now.
> >>
> >>Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
> >>
> >>+1 (binding), modulo that remaining question.
> >>
> >>* Verified signatures
> >>* Verified checksums for source and binary artefacts
> >>* Sanity checked jars on r.a.o.
> >>* Built from source
> >>* Deployed to 3 node secure cluster with NameNode HA
> >>* Verified HDFS web UIs
> >>* Tried out HDFS shell commands
> >>* Ran sample MapReduce jobs
> >>
> >>Thanks!
> >>
> >>
> >>----------------------------------------------------------------------
> >>From: Wangda Tan <wh...@gmail.com>
> >>Date: Monday, April 2, 2018 at 9:25 PM
> >>To: Arpit Agarwal <aa...@hortonworks.com>
> >>Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> >>Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> >>
> >>As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo?
> https://repository.apache.org/content/repositories/orgapachehadoop-1092
> >>
> >>Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
> >>
> >>Best,
> >>Wangda
> >>
> >>
> >>On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> >>Hi Arpit,
> >>
> >>Thanks for pointing out this.
> >>
> >>I just removed all .md5 files from artifacts. I found md5 checksums
> still exist in .mds files and I didn't remove them from .mds file because
> it is generated by create-release script and Apache guidance is "should
> not" instead of "must not". Please let me know if you think they need to be
> removed as well.
> >>
> >>- Wangda
> >>
> >>
> >>
> >>On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> >>Thanks for putting together this RC, Wangda.
> >>
> >>The guidance from Apache is to omit MD5s, specifically:
> >> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> >>
> >>https://www.apache.org/dev/release-distribution#sigs-and-sums
> >>
> >>
> >>
> >>
> >>On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
> >>
> >>Hi Gera,
> >>
> >>It's my bad, I thought only src/bin tarball is enough.
> >>
> >>I just uploaded all other things under artifact/ to
> >>http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> >>
> >>Please let me know if you have any other comments.
> >>
> >>Thanks,
> >>Wangda
> >>
> >>
> >>On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
> >>
> >>
> >>Thanks, Wangda!
> >>
> >>There are many more artifacts in previous votes, e.g., see
> >>http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> >>site tarball is missing.
> >>
> >>On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org>
> wrote:
> >>
> >>
> >>Thanks Wangda for initiating the release.
> >>
> >>I tested this RC built from source file.
> >>
> >>
> >> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> >>UI.
> >> - Below feature sanity is done
> >> - Application priority
> >> - Application timeout
> >> - Intra Queue preemption with priority based
> >> - DS based affinity tests to verify placement constraints.
> >> - Tested basic NodeLabel scenarios.
> >> - Added couple of labels to few of nodes and behavior is coming
> >> correct.
> >> - Verified old UI and new YARN UI for labels.
> >> - Submitted apps to labelled cluster and it works fine.
> >> - Also performed few cli commands related to nodelabel.
> >> - Test basic HA cases and seems correct.
> >> - Tested new YARN UI . All pages are getting loaded correctly.
> >>
> >>
> >>- Sunil
> >>
> >>On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
> >>
> >>
> >>Hi folks,
> >>
> >>Thanks to the many who helped with this release since Dec 2017 [1].
> >>We've
> >>
> >>created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >>
> >>http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >>
> >>The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> >>16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >>
> >>The maven artifacts are available via http://repository.apache.org at
> >>https://repository.apache.org/content/repositories/
> >>orgapachehadoop-1090/
> >>
> >>This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >>
> >>3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> >>include the first class GPU/FPGA support on YARN, Native services,
> >>Support
> >>
> >>rich placement constraints in YARN, S3-related enhancements, allow HDFS
> >>block replicas to be provided by an external storage system, etc.
> >>
> >>For 3.1.0 RC0 vote discussion, please see [3].
> >>
> >>We’d like to use this as a starting release for 3.1.x [1], depending on
> >>how
> >>
> >>it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> >>as
> >>
> >>the stable release.
> >>
> >>We have done testing with a pseudo cluster:
> >>- Ran distributed job.
> >>- GPU scheduling/isolation.
> >>- Placement constraints (intra-application anti-affinity) by using
> >>distributed shell.
> >>
> >>My +1 to start.
> >>
> >>Best,
> >>Wangda/Vinod
> >>
> >>[1]
> >>
> >>https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> >>c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
> >>
> >>[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> >>AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> >>BY
> >>
> >>fixVersion ASC
> >>[3]
> >>
> >>https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> >>2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>---------------------------------------------------------------------
> >>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >>For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Hanisha Koneru <hk...@hortonworks.com>.
Correction: My vote is NON-BINDING. Sorry for the confusion.
Thanks,
Hanisha
On 4/3/18, 11:40 AM, "Hanisha Koneru" <hk...@hortonworks.com> wrote:
>Thanks Wangda for putting up the RC for 3.1.0.
>
>+1 (binding).
>
>Verified the following:
>- Built from source
>- Deployed binary to a 3-node docker cluster
>- Sanity checks
> - Basic dfs operations
> - MapReduce Wordcount & Grep
>
>
>Thanks,
>Hanisha
>
>
>
>
>
>
>
>
>
>On 4/3/18, 9:33 AM, "Arpit Agarwal" <aa...@hortonworks.com> wrote:
>
>>Thanks Wangda, I see the shaded jars now.
>>
>>Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
>>
>>+1 (binding), modulo that remaining question.
>>
>>* Verified signatures
>>* Verified checksums for source and binary artefacts
>>* Sanity checked jars on r.a.o.
>>* Built from source
>>* Deployed to 3 node secure cluster with NameNode HA
>>* Verified HDFS web UIs
>>* Tried out HDFS shell commands
>>* Ran sample MapReduce jobs
>>
>>Thanks!
>>
>>
>>----------------------------------------------------------------------
>>From: Wangda Tan <wh...@gmail.com>
>>Date: Monday, April 2, 2018 at 9:25 PM
>>To: Arpit Agarwal <aa...@hortonworks.com>
>>Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <hd...@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@apache.org>
>>Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>
>>As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092
>>
>>Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll another RC, just update Maven repo should be sufficient.
>>
>>Best,
>>Wangda
>>
>>
>>On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com> wrote:
>>Hi Arpit,
>>
>>Thanks for pointing out this.
>>
>>I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
>>
>>- Wangda
>>
>>
>>
>>On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <ma...@hortonworks.com> wrote:
>>Thanks for putting together this RC, Wangda.
>>
>>The guidance from Apache is to omit MD5s, specifically:
>> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>>https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>>
>>On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
>>
>>Hi Gera,
>>
>>It's my bad, I thought only src/bin tarball is enough.
>>
>>I just uploaded all other things under artifact/ to
>>http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>>Please let me know if you have any other comments.
>>
>>Thanks,
>>Wangda
>>
>>
>>On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com> wrote:
>>
>>
>>Thanks, Wangda!
>>
>>There are many more artifacts in previous votes, e.g., see
>>http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>>site tarball is missing.
>>
>>On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>>
>>
>>Thanks Wangda for initiating the release.
>>
>>I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>>UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>>- Sunil
>>
>>On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com> wrote:
>>
>>
>>Hi folks,
>>
>>Thanks to the many who helped with this release since Dec 2017 [1].
>>We've
>>
>>created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>>http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>>The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>>16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>>The maven artifacts are available via http://repository.apache.org at
>>https://repository.apache.org/content/repositories/
>>orgapachehadoop-1090/
>>
>>This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>>3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>>include the first class GPU/FPGA support on YARN, Native services,
>>Support
>>
>>rich placement constraints in YARN, S3-related enhancements, allow HDFS
>>block replicas to be provided by an external storage system, etc.
>>
>>For 3.1.0 RC0 vote discussion, please see [3].
>>
>>We’d like to use this as a starting release for 3.1.x [1], depending on
>>how
>>
>>it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>>as
>>
>>the stable release.
>>
>>We have done testing with a pseudo cluster:
>>- Ran distributed job.
>>- GPU scheduling/isolation.
>>- Placement constraints (intra-application anti-affinity) by using
>>distributed shell.
>>
>>My +1 to start.
>>
>>Best,
>>Wangda/Vinod
>>
>>[1]
>>
>>https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
>>
>>[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>>AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>>BY
>>
>>fixVersion ASC
>>[3]
>>
>>https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Hanisha Koneru <hk...@hortonworks.com>.
Correction: My vote is NON-BINDING. Sorry for the confusion.
Thanks,
Hanisha
On 4/3/18, 11:40 AM, "Hanisha Koneru" <hk...@hortonworks.com> wrote:
>Thanks Wangda for putting up the RC for 3.1.0.
>
>+1 (binding).
>
>Verified the following:
>- Built from source
>- Deployed binary to a 3-node docker cluster
>- Sanity checks
> - Basic dfs operations
> - MapReduce Wordcount & Grep
>
>
>Thanks,
>Hanisha
>
>
>
>
>
>
>
>
>
>On 4/3/18, 9:33 AM, "Arpit Agarwal" <aa...@hortonworks.com> wrote:
>
>>Thanks Wangda, I see the shaded jars now.
>>
>>Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
>>
>>+1 (binding), modulo that remaining question.
>>
>>* Verified signatures
>>* Verified checksums for source and binary artefacts
>>* Sanity checked jars on r.a.o.
>>* Built from source
>>* Deployed to 3 node secure cluster with NameNode HA
>>* Verified HDFS web UIs
>>* Tried out HDFS shell commands
>>* Ran sample MapReduce jobs
>>
>>Thanks!
>>
>>
>>----------------------------------------------------------------------
>>From: Wangda Tan <wh...@gmail.com>
>>Date: Monday, April 2, 2018 at 9:25 PM
>>To: Arpit Agarwal <aa...@hortonworks.com>
>>Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <hd...@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@apache.org>
>>Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>
>>As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092
>>
>>Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll another RC, just update Maven repo should be sufficient.
>>
>>Best,
>>Wangda
>>
>>
>>On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com> wrote:
>>Hi Arpit,
>>
>>Thanks for pointing out this.
>>
>>I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
>>
>>- Wangda
>>
>>
>>
>>On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <ma...@hortonworks.com> wrote:
>>Thanks for putting together this RC, Wangda.
>>
>>The guidance from Apache is to omit MD5s, specifically:
>> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>>https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>>
>>On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
>>
>>Hi Gera,
>>
>>It's my bad, I thought only src/bin tarball is enough.
>>
>>I just uploaded all other things under artifact/ to
>>http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>>Please let me know if you have any other comments.
>>
>>Thanks,
>>Wangda
>>
>>
>>On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com> wrote:
>>
>>
>>Thanks, Wangda!
>>
>>There are many more artifacts in previous votes, e.g., see
>>http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>>site tarball is missing.
>>
>>On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>>
>>
>>Thanks Wangda for initiating the release.
>>
>>I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>>UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>>- Sunil
>>
>>On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com> wrote:
>>
>>
>>Hi folks,
>>
>>Thanks to the many who helped with this release since Dec 2017 [1].
>>We've
>>
>>created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>>http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>>The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>>16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>>The maven artifacts are available via http://repository.apache.org at
>>https://repository.apache.org/content/repositories/
>>orgapachehadoop-1090/
>>
>>This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>>3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>>include the first class GPU/FPGA support on YARN, Native services,
>>Support
>>
>>rich placement constraints in YARN, S3-related enhancements, allow HDFS
>>block replicas to be provided by an external storage system, etc.
>>
>>For 3.1.0 RC0 vote discussion, please see [3].
>>
>>We’d like to use this as a starting release for 3.1.x [1], depending on
>>how
>>
>>it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>>as
>>
>>the stable release.
>>
>>We have done testing with a pseudo cluster:
>>- Ran distributed job.
>>- GPU scheduling/isolation.
>>- Placement constraints (intra-application anti-affinity) by using
>>distributed shell.
>>
>>My +1 to start.
>>
>>Best,
>>Wangda/Vinod
>>
>>[1]
>>
>>https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
>>
>>[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>>AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>>BY
>>
>>fixVersion ASC
>>[3]
>>
>>https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Hanisha Koneru <hk...@hortonworks.com>.
Correction: My vote is NON-BINDING. Sorry for the confusion.
Thanks,
Hanisha
On 4/3/18, 11:40 AM, "Hanisha Koneru" <hk...@hortonworks.com> wrote:
>Thanks Wangda for putting up the RC for 3.1.0.
>
>+1 (binding).
>
>Verified the following:
>- Built from source
>- Deployed binary to a 3-node docker cluster
>- Sanity checks
> - Basic dfs operations
> - MapReduce Wordcount & Grep
>
>
>Thanks,
>Hanisha
>
>
>
>
>
>
>
>
>
>On 4/3/18, 9:33 AM, "Arpit Agarwal" <aa...@hortonworks.com> wrote:
>
>>Thanks Wangda, I see the shaded jars now.
>>
>>Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
>>
>>+1 (binding), modulo that remaining question.
>>
>>* Verified signatures
>>* Verified checksums for source and binary artefacts
>>* Sanity checked jars on r.a.o.
>>* Built from source
>>* Deployed to 3 node secure cluster with NameNode HA
>>* Verified HDFS web UIs
>>* Tried out HDFS shell commands
>>* Ran sample MapReduce jobs
>>
>>Thanks!
>>
>>
>>----------------------------------------------------------------------
>>From: Wangda Tan <wh...@gmail.com>
>>Date: Monday, April 2, 2018 at 9:25 PM
>>To: Arpit Agarwal <aa...@hortonworks.com>
>>Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <hd...@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@apache.org>
>>Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>
>>As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092
>>
>>Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll another RC, just update Maven repo should be sufficient.
>>
>>Best,
>>Wangda
>>
>>
>>On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com> wrote:
>>Hi Arpit,
>>
>>Thanks for pointing out this.
>>
>>I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
>>
>>- Wangda
>>
>>
>>
>>On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <ma...@hortonworks.com> wrote:
>>Thanks for putting together this RC, Wangda.
>>
>>The guidance from Apache is to omit MD5s, specifically:
>> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>>https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>>
>>On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
>>
>>Hi Gera,
>>
>>It's my bad, I thought only src/bin tarball is enough.
>>
>>I just uploaded all other things under artifact/ to
>>http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>>Please let me know if you have any other comments.
>>
>>Thanks,
>>Wangda
>>
>>
>>On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com> wrote:
>>
>>
>>Thanks, Wangda!
>>
>>There are many more artifacts in previous votes, e.g., see
>>http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>>site tarball is missing.
>>
>>On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>>
>>
>>Thanks Wangda for initiating the release.
>>
>>I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>>UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>>- Sunil
>>
>>On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com> wrote:
>>
>>
>>Hi folks,
>>
>>Thanks to the many who helped with this release since Dec 2017 [1].
>>We've
>>
>>created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>>http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>>The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>>16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>>The maven artifacts are available via http://repository.apache.org at
>>https://repository.apache.org/content/repositories/
>>orgapachehadoop-1090/
>>
>>This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>>3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>>include the first class GPU/FPGA support on YARN, Native services,
>>Support
>>
>>rich placement constraints in YARN, S3-related enhancements, allow HDFS
>>block replicas to be provided by an external storage system, etc.
>>
>>For 3.1.0 RC0 vote discussion, please see [3].
>>
>>We’d like to use this as a starting release for 3.1.x [1], depending on
>>how
>>
>>it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>>as
>>
>>the stable release.
>>
>>We have done testing with a pseudo cluster:
>>- Ran distributed job.
>>- GPU scheduling/isolation.
>>- Placement constraints (intra-application anti-affinity) by using
>>distributed shell.
>>
>>My +1 to start.
>>
>>Best,
>>Wangda/Vinod
>>
>>[1]
>>
>>https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
>>
>>[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>>AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>>BY
>>
>>fixVersion ASC
>>[3]
>>
>>https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Hanisha Koneru <hk...@hortonworks.com>.
Correction: My vote is NON-BINDING. Sorry for the confusion.
Thanks,
Hanisha
On 4/3/18, 11:40 AM, "Hanisha Koneru" <hk...@hortonworks.com> wrote:
>Thanks Wangda for putting up the RC for 3.1.0.
>
>+1 (binding).
>
>Verified the following:
>- Built from source
>- Deployed binary to a 3-node docker cluster
>- Sanity checks
> - Basic dfs operations
> - MapReduce Wordcount & Grep
>
>
>Thanks,
>Hanisha
>
>
>
>
>
>
>
>
>
>On 4/3/18, 9:33 AM, "Arpit Agarwal" <aa...@hortonworks.com> wrote:
>
>>Thanks Wangda, I see the shaded jars now.
>>
>>Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
>>
>>+1 (binding), modulo that remaining question.
>>
>>* Verified signatures
>>* Verified checksums for source and binary artefacts
>>* Sanity checked jars on r.a.o.
>>* Built from source
>>* Deployed to 3 node secure cluster with NameNode HA
>>* Verified HDFS web UIs
>>* Tried out HDFS shell commands
>>* Ran sample MapReduce jobs
>>
>>Thanks!
>>
>>
>>----------------------------------------------------------------------
>>From: Wangda Tan <wh...@gmail.com>
>>Date: Monday, April 2, 2018 at 9:25 PM
>>To: Arpit Agarwal <aa...@hortonworks.com>
>>Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <hd...@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@apache.org>
>>Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>
>>As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092
>>
>>Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll another RC, just update Maven repo should be sufficient.
>>
>>Best,
>>Wangda
>>
>>
>>On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com> wrote:
>>Hi Arpit,
>>
>>Thanks for pointing out this.
>>
>>I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
>>
>>- Wangda
>>
>>
>>
>>On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <ma...@hortonworks.com> wrote:
>>Thanks for putting together this RC, Wangda.
>>
>>The guidance from Apache is to omit MD5s, specifically:
>> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>>https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>>
>>On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
>>
>>Hi Gera,
>>
>>It's my bad, I thought only src/bin tarball is enough.
>>
>>I just uploaded all other things under artifact/ to
>>http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>>Please let me know if you have any other comments.
>>
>>Thanks,
>>Wangda
>>
>>
>>On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com> wrote:
>>
>>
>>Thanks, Wangda!
>>
>>There are many more artifacts in previous votes, e.g., see
>>http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>>site tarball is missing.
>>
>>On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>>
>>
>>Thanks Wangda for initiating the release.
>>
>>I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>>UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>>- Sunil
>>
>>On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com> wrote:
>>
>>
>>Hi folks,
>>
>>Thanks to the many who helped with this release since Dec 2017 [1].
>>We've
>>
>>created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>>http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>>The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>>16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>>The maven artifacts are available via http://repository.apache.org at
>>https://repository.apache.org/content/repositories/
>>orgapachehadoop-1090/
>>
>>This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>>3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>>include the first class GPU/FPGA support on YARN, Native services,
>>Support
>>
>>rich placement constraints in YARN, S3-related enhancements, allow HDFS
>>block replicas to be provided by an external storage system, etc.
>>
>>For 3.1.0 RC0 vote discussion, please see [3].
>>
>>We’d like to use this as a starting release for 3.1.x [1], depending on
>>how
>>
>>it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>>as
>>
>>the stable release.
>>
>>We have done testing with a pseudo cluster:
>>- Ran distributed job.
>>- GPU scheduling/isolation.
>>- Placement constraints (intra-application anti-affinity) by using
>>distributed shell.
>>
>>My +1 to start.
>>
>>Best,
>>Wangda/Vinod
>>
>>[1]
>>
>>https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
>>
>>[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>>AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>>BY
>>
>>fixVersion ASC
>>[3]
>>
>>https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Hanisha Koneru <hk...@hortonworks.com>.
Thanks Wangda for putting up the RC for 3.1.0.
+1 (binding).
Verified the following:
- Built from source
- Deployed binary to a 3-node docker cluster
- Sanity checks
- Basic dfs operations
- MapReduce Wordcount & Grep
Thanks,
Hanisha
On 4/3/18, 9:33 AM, "Arpit Agarwal" <aa...@hortonworks.com> wrote:
>Thanks Wangda, I see the shaded jars now.
>
>Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
>
>+1 (binding), modulo that remaining question.
>
>* Verified signatures
>* Verified checksums for source and binary artefacts
>* Sanity checked jars on r.a.o.
>* Built from source
>* Deployed to 3 node secure cluster with NameNode HA
>* Verified HDFS web UIs
>* Tried out HDFS shell commands
>* Ran sample MapReduce jobs
>
>Thanks!
>
>
>----------------------------------------------------------------------
>From: Wangda Tan <wh...@gmail.com>
>Date: Monday, April 2, 2018 at 9:25 PM
>To: Arpit Agarwal <aa...@hortonworks.com>
>Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <hd...@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@apache.org>
>Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
>As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092
>
>Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll another RC, just update Maven repo should be sufficient.
>
>Best,
>Wangda
>
>
>On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com> wrote:
>Hi Arpit,
>
>Thanks for pointing out this.
>
>I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
>
>- Wangda
>
>
>
>On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <ma...@hortonworks.com> wrote:
>Thanks for putting together this RC, Wangda.
>
>The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
>https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
>On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
>
>Hi Gera,
>
>It's my bad, I thought only src/bin tarball is enough.
>
>I just uploaded all other things under artifact/ to
>http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
>Please let me know if you have any other comments.
>
>Thanks,
>Wangda
>
>
>On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com> wrote:
>
>
>Thanks, Wangda!
>
>There are many more artifacts in previous votes, e.g., see
>http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>site tarball is missing.
>
>On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
>Thanks Wangda for initiating the release.
>
>I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
>- Sunil
>
>On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com> wrote:
>
>
>Hi folks,
>
>Thanks to the many who helped with this release since Dec 2017 [1].
>We've
>
>created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
>http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
>The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
>The maven artifacts are available via http://repository.apache.org at
>https://repository.apache.org/content/repositories/
>orgapachehadoop-1090/
>
>This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
>3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>include the first class GPU/FPGA support on YARN, Native services,
>Support
>
>rich placement constraints in YARN, S3-related enhancements, allow HDFS
>block replicas to be provided by an external storage system, etc.
>
>For 3.1.0 RC0 vote discussion, please see [3].
>
>We’d like to use this as a starting release for 3.1.x [1], depending on
>how
>
>it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>as
>
>the stable release.
>
>We have done testing with a pseudo cluster:
>- Ran distributed job.
>- GPU scheduling/isolation.
>- Placement constraints (intra-application anti-affinity) by using
>distributed shell.
>
>My +1 to start.
>
>Best,
>Wangda/Vinod
>
>[1]
>
>https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
>
>[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>BY
>
>fixVersion ASC
>[3]
>
>https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
>
>
>
>
>
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Hanisha Koneru <hk...@hortonworks.com>.
Thanks Wangda for putting up the RC for 3.1.0.
+1 (binding).
Verified the following:
- Built from source
- Deployed binary to a 3-node docker cluster
- Sanity checks
- Basic dfs operations
- MapReduce Wordcount & Grep
Thanks,
Hanisha
On 4/3/18, 9:33 AM, "Arpit Agarwal" <aa...@hortonworks.com> wrote:
>Thanks Wangda, I see the shaded jars now.
>
>Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
>
>+1 (binding), modulo that remaining question.
>
>* Verified signatures
>* Verified checksums for source and binary artefacts
>* Sanity checked jars on r.a.o.
>* Built from source
>* Deployed to 3 node secure cluster with NameNode HA
>* Verified HDFS web UIs
>* Tried out HDFS shell commands
>* Ran sample MapReduce jobs
>
>Thanks!
>
>
>----------------------------------------------------------------------
>From: Wangda Tan <wh...@gmail.com>
>Date: Monday, April 2, 2018 at 9:25 PM
>To: Arpit Agarwal <aa...@hortonworks.com>
>Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <hd...@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@apache.org>
>Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
>As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092
>
>Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll another RC, just update Maven repo should be sufficient.
>
>Best,
>Wangda
>
>
>On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com> wrote:
>Hi Arpit,
>
>Thanks for pointing out this.
>
>I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
>
>- Wangda
>
>
>
>On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <ma...@hortonworks.com> wrote:
>Thanks for putting together this RC, Wangda.
>
>The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
>https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
>On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
>
>Hi Gera,
>
>It's my bad, I thought only src/bin tarball is enough.
>
>I just uploaded all other things under artifact/ to
>http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
>Please let me know if you have any other comments.
>
>Thanks,
>Wangda
>
>
>On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com> wrote:
>
>
>Thanks, Wangda!
>
>There are many more artifacts in previous votes, e.g., see
>http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>site tarball is missing.
>
>On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
>Thanks Wangda for initiating the release.
>
>I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
>- Sunil
>
>On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com> wrote:
>
>
>Hi folks,
>
>Thanks to the many who helped with this release since Dec 2017 [1].
>We've
>
>created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
>http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
>The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
>The maven artifacts are available via http://repository.apache.org at
>https://repository.apache.org/content/repositories/
>orgapachehadoop-1090/
>
>This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
>3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>include the first class GPU/FPGA support on YARN, Native services,
>Support
>
>rich placement constraints in YARN, S3-related enhancements, allow HDFS
>block replicas to be provided by an external storage system, etc.
>
>For 3.1.0 RC0 vote discussion, please see [3].
>
>We’d like to use this as a starting release for 3.1.x [1], depending on
>how
>
>it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>as
>
>the stable release.
>
>We have done testing with a pseudo cluster:
>- Ran distributed job.
>- GPU scheduling/isolation.
>- Placement constraints (intra-application anti-affinity) by using
>distributed shell.
>
>My +1 to start.
>
>Best,
>Wangda/Vinod
>
>[1]
>
>https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
>
>[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>BY
>
>fixVersion ASC
>[3]
>
>https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
>
>
>
>
>
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Hanisha Koneru <hk...@hortonworks.com>.
Thanks Wangda for putting up the RC for 3.1.0.
+1 (binding).
Verified the following:
- Built from source
- Deployed binary to a 3-node docker cluster
- Sanity checks
- Basic dfs operations
- MapReduce Wordcount & Grep
Thanks,
Hanisha
On 4/3/18, 9:33 AM, "Arpit Agarwal" <aa...@hortonworks.com> wrote:
>Thanks Wangda, I see the shaded jars now.
>
>Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
>
>+1 (binding), modulo that remaining question.
>
>* Verified signatures
>* Verified checksums for source and binary artefacts
>* Sanity checked jars on r.a.o.
>* Built from source
>* Deployed to 3 node secure cluster with NameNode HA
>* Verified HDFS web UIs
>* Tried out HDFS shell commands
>* Ran sample MapReduce jobs
>
>Thanks!
>
>
>----------------------------------------------------------------------
>From: Wangda Tan <wh...@gmail.com>
>Date: Monday, April 2, 2018 at 9:25 PM
>To: Arpit Agarwal <aa...@hortonworks.com>
>Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <hd...@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@apache.org>
>Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
>As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092
>
>Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll another RC, just update Maven repo should be sufficient.
>
>Best,
>Wangda
>
>
>On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com> wrote:
>Hi Arpit,
>
>Thanks for pointing out this.
>
>I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
>
>- Wangda
>
>
>
>On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <ma...@hortonworks.com> wrote:
>Thanks for putting together this RC, Wangda.
>
>The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
>https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
>On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
>
>Hi Gera,
>
>It's my bad, I thought only src/bin tarball is enough.
>
>I just uploaded all other things under artifact/ to
>http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
>Please let me know if you have any other comments.
>
>Thanks,
>Wangda
>
>
>On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com> wrote:
>
>
>Thanks, Wangda!
>
>There are many more artifacts in previous votes, e.g., see
>http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>site tarball is missing.
>
>On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
>Thanks Wangda for initiating the release.
>
>I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
>- Sunil
>
>On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com> wrote:
>
>
>Hi folks,
>
>Thanks to the many who helped with this release since Dec 2017 [1].
>We've
>
>created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
>http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
>The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
>The maven artifacts are available via http://repository.apache.org at
>https://repository.apache.org/content/repositories/
>orgapachehadoop-1090/
>
>This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
>3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>include the first class GPU/FPGA support on YARN, Native services,
>Support
>
>rich placement constraints in YARN, S3-related enhancements, allow HDFS
>block replicas to be provided by an external storage system, etc.
>
>For 3.1.0 RC0 vote discussion, please see [3].
>
>We’d like to use this as a starting release for 3.1.x [1], depending on
>how
>
>it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>as
>
>the stable release.
>
>We have done testing with a pseudo cluster:
>- Ran distributed job.
>- GPU scheduling/isolation.
>- Placement constraints (intra-application anti-affinity) by using
>distributed shell.
>
>My +1 to start.
>
>Best,
>Wangda/Vinod
>
>[1]
>
>https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
>
>[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>BY
>
>fixVersion ASC
>[3]
>
>https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
>
>
>
>
>
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Hanisha Koneru <hk...@hortonworks.com>.
Thanks Wangda for putting up the RC for 3.1.0.
+1 (binding).
Verified the following:
- Built from source
- Deployed binary to a 3-node docker cluster
- Sanity checks
- Basic dfs operations
- MapReduce Wordcount & Grep
Thanks,
Hanisha
On 4/3/18, 9:33 AM, "Arpit Agarwal" <aa...@hortonworks.com> wrote:
>Thanks Wangda, I see the shaded jars now.
>
>Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
>
>+1 (binding), modulo that remaining question.
>
>* Verified signatures
>* Verified checksums for source and binary artefacts
>* Sanity checked jars on r.a.o.
>* Built from source
>* Deployed to 3 node secure cluster with NameNode HA
>* Verified HDFS web UIs
>* Tried out HDFS shell commands
>* Ran sample MapReduce jobs
>
>Thanks!
>
>
>----------------------------------------------------------------------
>From: Wangda Tan <wh...@gmail.com>
>Date: Monday, April 2, 2018 at 9:25 PM
>To: Arpit Agarwal <aa...@hortonworks.com>
>Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <hd...@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@apache.org>
>Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
>As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092
>
>Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll another RC, just update Maven repo should be sufficient.
>
>Best,
>Wangda
>
>
>On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com> wrote:
>Hi Arpit,
>
>Thanks for pointing out this.
>
>I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
>
>- Wangda
>
>
>
>On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <ma...@hortonworks.com> wrote:
>Thanks for putting together this RC, Wangda.
>
>The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
>https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
>On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
>
>Hi Gera,
>
>It's my bad, I thought only src/bin tarball is enough.
>
>I just uploaded all other things under artifact/ to
>http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
>Please let me know if you have any other comments.
>
>Thanks,
>Wangda
>
>
>On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com> wrote:
>
>
>Thanks, Wangda!
>
>There are many more artifacts in previous votes, e.g., see
>http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>site tarball is missing.
>
>On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
>Thanks Wangda for initiating the release.
>
>I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
>- Sunil
>
>On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com> wrote:
>
>
>Hi folks,
>
>Thanks to the many who helped with this release since Dec 2017 [1].
>We've
>
>created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
>http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
>The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
>The maven artifacts are available via http://repository.apache.org at
>https://repository.apache.org/content/repositories/
>orgapachehadoop-1090/
>
>This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
>3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>include the first class GPU/FPGA support on YARN, Native services,
>Support
>
>rich placement constraints in YARN, S3-related enhancements, allow HDFS
>block replicas to be provided by an external storage system, etc.
>
>For 3.1.0 RC0 vote discussion, please see [3].
>
>We’d like to use this as a starting release for 3.1.x [1], depending on
>how
>
>it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>as
>
>the stable release.
>
>We have done testing with a pseudo cluster:
>- Ran distributed job.
>- GPU scheduling/isolation.
>- Placement constraints (intra-application anti-affinity) by using
>distributed shell.
>
>My +1 to start.
>
>Best,
>Wangda/Vinod
>
>[1]
>
>https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
>
>[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>BY
>
>fixVersion ASC
>[3]
>
>https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
>
>
>
>
>
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Iñigo Goiri <el...@gmail.com>.
+1 (non binding)
* Deployed with 4 subclusters with HDFS Router-based federation.
* Executed DistCp across subclusters through the Router
* Checked documentation and tgz
On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
> > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > HI Arpit,
> >
> > I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com
> <ma...@hortonworks.com>> wrote:
> > Thanks Wangda, I see the shaded jars now.
> >
> > Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
> >
> > +1 (binding), modulo that remaining question.
> >
> > * Verified signatures
> > * Verified checksums for source and binary artefacts
> > * Sanity checked jars on r.a.o.
> > * Built from source
> > * Deployed to 3 node secure cluster with NameNode HA
> > * Verified HDFS web UIs
> > * Tried out HDFS shell commands
> > * Ran sample MapReduce jobs
> >
> > Thanks!
> >
> >
> > ----------------------------------------------------------------------
> > From: Wangda Tan <wheeleast@gmail.com <ma...@gmail.com>>
> > Date: Monday, April 2, 2018 at 9:25 PM
> > To: Arpit Agarwal <aagarwal@hortonworks.com <mailto:
> aagarwal@hortonworks.com>>
> > Cc: Gera Shegalov <gera05@gmail.com <ma...@gmail.com>>, Sunil G
> <sunilg@apache.org <ma...@apache.org>>, "
> yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>" <
> yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hdfs-dev
> <hdfs-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hadoop
> Common <common-dev@hadoop.apache.org <ma...@hadoop.apache.org>>,
> "mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>"
> <mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>>,
> Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>>
> > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> >
> > As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092 <https://repository.apache.org/content/repositories/
> orgapachehadoop-1092>
> >
> > Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
> >
> > Best,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com
> <ma...@gmail.com>> wrote:
> > Hi Arpit,
> >
> > Thanks for pointing out this.
> >
> > I just removed all .md5 files from artifacts. I found md5 checksums
> still exist in .mds files and I didn't remove them from .mds file because
> it is generated by create-release script and Apache guidance is "should
> not" instead of "must not". Please let me know if you think they need to be
> removed as well.
> >
> > - Wangda
> >
> >
> >
> > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com <ma...@hortonworks.com>> wrote:
> > Thanks for putting together this RC, Wangda.
> >
> > The guidance from Apache is to omit MD5s, specifically:
> > > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> >
> > https://www.apache.org/dev/release-distribution#sigs-and-sums <
> https://www.apache.org/dev/release-distribution#sigs-and-sums>
> >
> >
> >
> >
> > On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheeleast@gmail.com
> <ma...@gmail.com>> wrote:
> >
> > Hi Gera,
> >
> > It's my bad, I thought only src/bin tarball is enough.
> >
> > I just uploaded all other things under artifact/ to
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>
> >
> > Please let me know if you have any other comments.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:gera05@gmail.com
> <ma...@gmail.com>> wrote:
> >
> >
> > Thanks, Wangda!
> >
> > There are many more artifacts in previous votes, e.g., see
> > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ <
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/> . Among others the
> > site tarball is missing.
> >
> > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sunilg@apache.org
> <ma...@apache.org>> wrote:
> >
> >
> > Thanks Wangda for initiating the release.
> >
> > I tested this RC built from source file.
> >
> >
> > - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> > UI.
> > - Below feature sanity is done
> > - Application priority
> > - Application timeout
> > - Intra Queue preemption with priority based
> > - DS based affinity tests to verify placement constraints.
> > - Tested basic NodeLabel scenarios.
> > - Added couple of labels to few of nodes and behavior is coming
> > correct.
> > - Verified old UI and new YARN UI for labels.
> > - Submitted apps to labelled cluster and it works fine.
> > - Also performed few cli commands related to nodelabel.
> > - Test basic HA cases and seems correct.
> > - Tested new YARN UI . All pages are getting loaded correctly.
> >
> >
> > - Sunil
> >
> > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com
> <ma...@gmail.com>> wrote:
> >
> >
> > Hi folks,
> >
> > Thanks to the many who helped with this release since Dec 2017 [1].
> > We've
> >
> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1 <
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1>
> >
> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >
> > The maven artifacts are available via http://repository.apache.org <
> http://repository.apache.org/> at
> > https://repository.apache.org/content/repositories/ <
> https://repository.apache.org/content/repositories/>
> > orgapachehadoop-1090/
> >
> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >
> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > include the first class GPU/FPGA support on YARN, Native services,
> > Support
> >
> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > block replicas to be provided by an external storage system, etc.
> >
> > For 3.1.0 RC0 vote discussion, please see [3].
> >
> > We’d like to use this as a starting release for 3.1.x [1], depending on
> > how
> >
> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> > as
> >
> > the stable release.
> >
> > We have done testing with a pseudo cluster:
> > - Ran distributed job.
> > - GPU scheduling/isolation.
> > - Placement constraints (intra-application anti-affinity) by using
> > distributed shell.
> >
> > My +1 to start.
> >
> > Best,
> > Wangda/Vinod
> >
> > [1]
> >
> > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b <
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b>
> > c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org <
> http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> > BY
> >
> > fixVersion ASC
> > [3]
> >
> > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7 <
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7>
> > 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org <
> http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Shane Kumpf <sh...@gmail.com>.
Thanks for putting together the RC, Wangda! Sorry for the late addition.
+1 (non-binding)
- Verified checksums and signatures of the artifacts
- Deployed a pseudo distributed cluster using the binary tgz, source tgz,
and git tag
- Ran basic hdfs commands
- Ran pi, sleep, distributed shell, and a sleeper service using the default
and docker runtimes
- Ran dshell using the default and docker runtimes in a secure cluster
- Verified docker runtime and privileged containers can be disabled
- Verified YARN and HDFS UI's
- Validated the change log against the commits, all discrepancies are
documented or expected
- Basic validation of Registry DNS lookups
On Thu, Apr 5, 2018 at 12:26 AM, Naganarasimha Garla <
naganarasimha_gr@apache.org> wrote:
> +1 (binding)
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Built from source
> * Deployed a pseudo Hadoop cluster
> * Verified YARN & HDFS web UIs
> * Ran sample MapReduce jobs
> * Tested NodeLabels
>
> Regards,
> + Naga
>
> On Thu, Apr 5, 2018 at 2:59 AM, Suma Shivaprasad <
> sumasai.shivaprasad@gmail.com> wrote:
>
> > +1 (non binding)
> >
> >
> > *Verified - User Group Queue mapping - Node labels with New UI- Dynamic
> > queuesThanksSuma*
> >
> >
> > On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran <st...@hortonworks.com>
> > wrote:
> >
> > > that's "dangerously interesting". I think you are right, and I also
> think
> > > it'll just be the version files which get generated
> > >
> > > anyway, +1 binding
> > >
> > > * ran my new Hadoop-3 profile on spark (SPARK-23807), with the
> committer
> > > binding, then my downstream tests. All is well, provided you also have
> a
> > > spark hive JAR patched to accept hadoop 3 as a legitimate hadoop
> version.
> > > That's an ongoing issue in the Spark project. With that JAR on my CP my
> > > downstream tests were all happy (yesterday)
> > >
> > > * today the staging files seem to be missing, at least maven is unable
> to
> > > find them even when I turn the spark snapshots-and-staging profile on.
> > > That'll be the maven dist process at play, nothing else
> > >
> > > On 4 Apr 2018, at 04:13, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > > eeleast@gmail.com>> wrote:
> > >
> > > Hi Vinod / Arpit,
> > >
> > > I checked following versions:
> > > - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
> > >
> > > Jars in maven repo [1] are *always* different from jars in the binary
> > > tarball [2]: (I only checked hadoop-yarn-api-version.jar)
> > >
> > > (Following numbers are sizes of the jar)
> > > 2.6.5:
> > > - Jar in Maven: 1896185
> > > - Jar in tarball: 1891485
> > >
> > > 2.7.5:
> > > - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> > > - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> > > (Even size is same, md5 is different)
> > >
> > > 2.8.3:
> > > - Jar in Maven: 2451433
> > > - Jar in tarball: 2438975
> > >
> > > 2.9.0:
> > > - Jar in Maven: 2791477
> > > - Jar in tarball: 2777789
> > >
> > > 3.0.1:
> > > - Jar in Maven: 2852604
> > > - Jar in tarball: 2851373
> > >
> > > I guess the differences come from our release process.
> > >
> > > Thanks,
> > > Wangda
> > >
> > > [1] Maven jars are downloaded from
> > > https://repository.apache.org/service/local/repositories/
> > > releases/content/org/apache/hadoop/hadoop-yarn-api/
> > > <version>/hadoop-yarn-api-<version>.jar
> > > [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/
> > common/
> > >
> > >
> > > On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> > > vinodkv@apache.org<ma...@apache.org>>
> > > wrote:
> > >
> > > We vote on the source code. The binaries are convenience artifacts.
> > >
> > > This is what I would do - (a) Just replace both the maven jars as well
> as
> > > the binaries to be consistent and correct. And then (b) Give a couple
> > more
> > > days for folks who tested on the binaries to reverify - I count one
> such
> > > clear vote as of now.
> > >
> > > Thanks
> > > +Vinod
> > >
> > >
> > > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > > eeleast@gmail.com>> wrote:
> > >
> > > HI Arpit,
> > >
> > > I think it won't match if we do rebuild. It should be fine as far as
> > > they're signed, correct? I don't see any policy doesn't allow this.
> > >
> > > Thanks,
> > > Wangda
> > >
> > >
> > > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <
> aagarwal@hortonworks.com<
> > > mailto:aagarwal@hortonworks.com>>
> > > wrote:
> > >
> > > Thanks Wangda, I see the shaded jars now.
> > >
> > > Are the repo jars required to be the same as the binary release? They
> > > don’t match right now, probably they got rebuilt.
> > >
> > > +1 (binding), modulo that remaining question.
> > >
> > > * Verified signatures
> > > * Verified checksums for source and binary artefacts
> > > * Sanity checked jars on r.a.o.
> > > * Built from source
> > > * Deployed to 3 node secure cluster with NameNode HA
> > > * Verified HDFS web UIs
> > > * Tried out HDFS shell commands
> > > * Ran sample MapReduce jobs
> > >
> > > Thanks!
> > >
> > >
> > > ----------------------------------------------------------------------
> > > From: Wangda Tan <wh...@gmail.com>>
> > > Date: Monday, April 2, 2018 at 9:25 PM
> > > To: Arpit Agarwal <aagarwal@hortonworks.com<mailto:
> aagarwal@hortonworks.
> > > com>>
> > > Cc: Gera Shegalov <ge...@gmail.com>>, Sunil
> G <
> > > sunilg@apache.org<ma...@apache.org>>, "
> > > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
> > > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>>,
> > Hdfs-dev <
> > > hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop
> > > Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.
> apache.org
> > > >>,
> > > "mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@
> hadoop.apache.org
> > >"
> > > <mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@
> hadoop.apache.org
> > >>,
> > > Vinod Kumar Vavilapalli <vinodkv@apache.org<mailto:vinodkv@apache.org
> >>
> > > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> > >
> > > As pointed by Arpit, the previously deployed shared jars are incorrect.
> > > Just redeployed jars and staged. @Arpit, could you please check the
> > updated
> > > Maven repo? https://repository.apache.org/content/repositories/
> > > orgapachehadoop-1092
> > >
> > > Since the jars inside binary tarballs are correct (
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> > > need roll another RC, just update Maven repo should be sufficient.
> > >
> > > Best,
> > > Wangda
> > >
> > >
> > > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com
> >
> > > wrote:
> > > Hi Arpit,
> > >
> > > Thanks for pointing out this.
> > >
> > > I just removed all .md5 files from artifacts. I found md5 checksums
> still
> > > exist in .mds files and I didn't remove them from .mds file because it
> is
> > > generated by create-release script and Apache guidance is "should not"
> > > instead of "must not". Please let me know if you think they need to be
> > > removed as well.
> > >
> > > - Wangda
> > >
> > >
> > >
> > > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> > > aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
> > > Thanks for putting together this RC, Wangda.
> > >
> > > The guidance from Apache is to omit MD5s, specifically:
> > > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> > >
> > > https://www.apache.org/dev/release-distribution#sigs-and-sums
> > >
> > >
> > >
> > >
> > > On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> > > wrote:
> > >
> > > Hi Gera,
> > >
> > > It's my bad, I thought only src/bin tarball is enough.
> > >
> > > I just uploaded all other things under artifact/ to
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> > >
> > > Please let me know if you have any other comments.
> > >
> > > Thanks,
> > > Wangda
> > >
> > >
> > > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:
> gera05@gmail.com>
> > > wrote:
> > >
> > >
> > > Thanks, Wangda!
> > >
> > > There are many more artifacts in previous votes, e.g., see
> > > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others
> the
> > > site tarball is missing.
> > >
> > > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org>
> > wrote:
> > >
> > >
> > > Thanks Wangda for initiating the release.
> > >
> > > I tested this RC built from source file.
> > >
> > >
> > > - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> > > UI.
> > > - Below feature sanity is done
> > > - Application priority
> > > - Application timeout
> > > - Intra Queue preemption with priority based
> > > - DS based affinity tests to verify placement constraints.
> > > - Tested basic NodeLabel scenarios.
> > > - Added couple of labels to few of nodes and behavior is coming
> > > correct.
> > > - Verified old UI and new YARN UI for labels.
> > > - Submitted apps to labelled cluster and it works fine.
> > > - Also performed few cli commands related to nodelabel.
> > > - Test basic HA cases and seems correct.
> > > - Tested new YARN UI . All pages are getting loaded correctly.
> > >
> > >
> > > - Sunil
> > >
> > > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com
> >
> > > wrote:
> > >
> > >
> > > Hi folks,
> > >
> > > Thanks to the many who helped with this release since Dec 2017 [1].
> > > We've
> > >
> > > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> > >
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> > >
> > > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> > >
> > > The maven artifacts are available via http://repository.apache.org<h
> > > ttp://repository.apache.org/> at
> > > https://repository.apache.org/content/repositories/
> > > orgapachehadoop-1090/
> > >
> > > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> > >
> > > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > > include the first class GPU/FPGA support on YARN, Native services,
> > > Support
> > >
> > > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > > block replicas to be provided by an external storage system, etc.
> > >
> > > For 3.1.0 RC0 vote discussion, please see [3].
> > >
> > > We’d like to use this as a starting release for 3.1.x [1], depending on
> > > how
> > >
> > > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> > > as
> > >
> > > the stable release.
> > >
> > > We have done testing with a pseudo cluster:
> > > - Ran distributed job.
> > > - GPU scheduling/isolation.
> > > - Placement constraints (intra-application anti-affinity) by using
> > > distributed shell.
> > >
> > > My +1 to start.
> > >
> > > Best,
> > > Wangda/Vinod
> > >
> > > [1]
> > >
> > > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> > > c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<
> > > http://3cyarn-dev.hadoop.apache.org/>
> > > <http://3cyarn-dev.hadoop.apache.org/>%3E
> > >
> > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.0)
> > > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> > > BY
> > >
> > > fixVersion ASC
> > > [3]
> > >
> > > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> > > 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<
> > > http://3cyarn-dev.hadoop.apache.org/>
> > > <http://3cyarn-dev.hadoop.apache.org/>%3E
> > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Shane Kumpf <sh...@gmail.com>.
Thanks for putting together the RC, Wangda! Sorry for the late addition.
+1 (non-binding)
- Verified checksums and signatures of the artifacts
- Deployed a pseudo distributed cluster using the binary tgz, source tgz,
and git tag
- Ran basic hdfs commands
- Ran pi, sleep, distributed shell, and a sleeper service using the default
and docker runtimes
- Ran dshell using the default and docker runtimes in a secure cluster
- Verified docker runtime and privileged containers can be disabled
- Verified YARN and HDFS UI's
- Validated the change log against the commits, all discrepancies are
documented or expected
- Basic validation of Registry DNS lookups
On Thu, Apr 5, 2018 at 12:26 AM, Naganarasimha Garla <
naganarasimha_gr@apache.org> wrote:
> +1 (binding)
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Built from source
> * Deployed a pseudo Hadoop cluster
> * Verified YARN & HDFS web UIs
> * Ran sample MapReduce jobs
> * Tested NodeLabels
>
> Regards,
> + Naga
>
> On Thu, Apr 5, 2018 at 2:59 AM, Suma Shivaprasad <
> sumasai.shivaprasad@gmail.com> wrote:
>
> > +1 (non binding)
> >
> >
> > *Verified - User Group Queue mapping - Node labels with New UI- Dynamic
> > queuesThanksSuma*
> >
> >
> > On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran <st...@hortonworks.com>
> > wrote:
> >
> > > that's "dangerously interesting". I think you are right, and I also
> think
> > > it'll just be the version files which get generated
> > >
> > > anyway, +1 binding
> > >
> > > * ran my new Hadoop-3 profile on spark (SPARK-23807), with the
> committer
> > > binding, then my downstream tests. All is well, provided you also have
> a
> > > spark hive JAR patched to accept hadoop 3 as a legitimate hadoop
> version.
> > > That's an ongoing issue in the Spark project. With that JAR on my CP my
> > > downstream tests were all happy (yesterday)
> > >
> > > * today the staging files seem to be missing, at least maven is unable
> to
> > > find them even when I turn the spark snapshots-and-staging profile on.
> > > That'll be the maven dist process at play, nothing else
> > >
> > > On 4 Apr 2018, at 04:13, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > > eeleast@gmail.com>> wrote:
> > >
> > > Hi Vinod / Arpit,
> > >
> > > I checked following versions:
> > > - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
> > >
> > > Jars in maven repo [1] are *always* different from jars in the binary
> > > tarball [2]: (I only checked hadoop-yarn-api-version.jar)
> > >
> > > (Following numbers are sizes of the jar)
> > > 2.6.5:
> > > - Jar in Maven: 1896185
> > > - Jar in tarball: 1891485
> > >
> > > 2.7.5:
> > > - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> > > - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> > > (Even size is same, md5 is different)
> > >
> > > 2.8.3:
> > > - Jar in Maven: 2451433
> > > - Jar in tarball: 2438975
> > >
> > > 2.9.0:
> > > - Jar in Maven: 2791477
> > > - Jar in tarball: 2777789
> > >
> > > 3.0.1:
> > > - Jar in Maven: 2852604
> > > - Jar in tarball: 2851373
> > >
> > > I guess the differences come from our release process.
> > >
> > > Thanks,
> > > Wangda
> > >
> > > [1] Maven jars are downloaded from
> > > https://repository.apache.org/service/local/repositories/
> > > releases/content/org/apache/hadoop/hadoop-yarn-api/
> > > <version>/hadoop-yarn-api-<version>.jar
> > > [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/
> > common/
> > >
> > >
> > > On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> > > vinodkv@apache.org<ma...@apache.org>>
> > > wrote:
> > >
> > > We vote on the source code. The binaries are convenience artifacts.
> > >
> > > This is what I would do - (a) Just replace both the maven jars as well
> as
> > > the binaries to be consistent and correct. And then (b) Give a couple
> > more
> > > days for folks who tested on the binaries to reverify - I count one
> such
> > > clear vote as of now.
> > >
> > > Thanks
> > > +Vinod
> > >
> > >
> > > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > > eeleast@gmail.com>> wrote:
> > >
> > > HI Arpit,
> > >
> > > I think it won't match if we do rebuild. It should be fine as far as
> > > they're signed, correct? I don't see any policy doesn't allow this.
> > >
> > > Thanks,
> > > Wangda
> > >
> > >
> > > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <
> aagarwal@hortonworks.com<
> > > mailto:aagarwal@hortonworks.com>>
> > > wrote:
> > >
> > > Thanks Wangda, I see the shaded jars now.
> > >
> > > Are the repo jars required to be the same as the binary release? They
> > > don’t match right now, probably they got rebuilt.
> > >
> > > +1 (binding), modulo that remaining question.
> > >
> > > * Verified signatures
> > > * Verified checksums for source and binary artefacts
> > > * Sanity checked jars on r.a.o.
> > > * Built from source
> > > * Deployed to 3 node secure cluster with NameNode HA
> > > * Verified HDFS web UIs
> > > * Tried out HDFS shell commands
> > > * Ran sample MapReduce jobs
> > >
> > > Thanks!
> > >
> > >
> > > ----------------------------------------------------------------------
> > > From: Wangda Tan <wh...@gmail.com>>
> > > Date: Monday, April 2, 2018 at 9:25 PM
> > > To: Arpit Agarwal <aagarwal@hortonworks.com<mailto:
> aagarwal@hortonworks.
> > > com>>
> > > Cc: Gera Shegalov <ge...@gmail.com>>, Sunil
> G <
> > > sunilg@apache.org<ma...@apache.org>>, "
> > > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
> > > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>>,
> > Hdfs-dev <
> > > hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop
> > > Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.
> apache.org
> > > >>,
> > > "mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@
> hadoop.apache.org
> > >"
> > > <mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@
> hadoop.apache.org
> > >>,
> > > Vinod Kumar Vavilapalli <vinodkv@apache.org<mailto:vinodkv@apache.org
> >>
> > > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> > >
> > > As pointed by Arpit, the previously deployed shared jars are incorrect.
> > > Just redeployed jars and staged. @Arpit, could you please check the
> > updated
> > > Maven repo? https://repository.apache.org/content/repositories/
> > > orgapachehadoop-1092
> > >
> > > Since the jars inside binary tarballs are correct (
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> > > need roll another RC, just update Maven repo should be sufficient.
> > >
> > > Best,
> > > Wangda
> > >
> > >
> > > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com
> >
> > > wrote:
> > > Hi Arpit,
> > >
> > > Thanks for pointing out this.
> > >
> > > I just removed all .md5 files from artifacts. I found md5 checksums
> still
> > > exist in .mds files and I didn't remove them from .mds file because it
> is
> > > generated by create-release script and Apache guidance is "should not"
> > > instead of "must not". Please let me know if you think they need to be
> > > removed as well.
> > >
> > > - Wangda
> > >
> > >
> > >
> > > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> > > aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
> > > Thanks for putting together this RC, Wangda.
> > >
> > > The guidance from Apache is to omit MD5s, specifically:
> > > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> > >
> > > https://www.apache.org/dev/release-distribution#sigs-and-sums
> > >
> > >
> > >
> > >
> > > On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> > > wrote:
> > >
> > > Hi Gera,
> > >
> > > It's my bad, I thought only src/bin tarball is enough.
> > >
> > > I just uploaded all other things under artifact/ to
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> > >
> > > Please let me know if you have any other comments.
> > >
> > > Thanks,
> > > Wangda
> > >
> > >
> > > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:
> gera05@gmail.com>
> > > wrote:
> > >
> > >
> > > Thanks, Wangda!
> > >
> > > There are many more artifacts in previous votes, e.g., see
> > > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others
> the
> > > site tarball is missing.
> > >
> > > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org>
> > wrote:
> > >
> > >
> > > Thanks Wangda for initiating the release.
> > >
> > > I tested this RC built from source file.
> > >
> > >
> > > - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> > > UI.
> > > - Below feature sanity is done
> > > - Application priority
> > > - Application timeout
> > > - Intra Queue preemption with priority based
> > > - DS based affinity tests to verify placement constraints.
> > > - Tested basic NodeLabel scenarios.
> > > - Added couple of labels to few of nodes and behavior is coming
> > > correct.
> > > - Verified old UI and new YARN UI for labels.
> > > - Submitted apps to labelled cluster and it works fine.
> > > - Also performed few cli commands related to nodelabel.
> > > - Test basic HA cases and seems correct.
> > > - Tested new YARN UI . All pages are getting loaded correctly.
> > >
> > >
> > > - Sunil
> > >
> > > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com
> >
> > > wrote:
> > >
> > >
> > > Hi folks,
> > >
> > > Thanks to the many who helped with this release since Dec 2017 [1].
> > > We've
> > >
> > > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> > >
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> > >
> > > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> > >
> > > The maven artifacts are available via http://repository.apache.org<h
> > > ttp://repository.apache.org/> at
> > > https://repository.apache.org/content/repositories/
> > > orgapachehadoop-1090/
> > >
> > > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> > >
> > > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > > include the first class GPU/FPGA support on YARN, Native services,
> > > Support
> > >
> > > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > > block replicas to be provided by an external storage system, etc.
> > >
> > > For 3.1.0 RC0 vote discussion, please see [3].
> > >
> > > We’d like to use this as a starting release for 3.1.x [1], depending on
> > > how
> > >
> > > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> > > as
> > >
> > > the stable release.
> > >
> > > We have done testing with a pseudo cluster:
> > > - Ran distributed job.
> > > - GPU scheduling/isolation.
> > > - Placement constraints (intra-application anti-affinity) by using
> > > distributed shell.
> > >
> > > My +1 to start.
> > >
> > > Best,
> > > Wangda/Vinod
> > >
> > > [1]
> > >
> > > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> > > c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<
> > > http://3cyarn-dev.hadoop.apache.org/>
> > > <http://3cyarn-dev.hadoop.apache.org/>%3E
> > >
> > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.0)
> > > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> > > BY
> > >
> > > fixVersion ASC
> > > [3]
> > >
> > > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> > > 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<
> > > http://3cyarn-dev.hadoop.apache.org/>
> > > <http://3cyarn-dev.hadoop.apache.org/>%3E
> > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Shane Kumpf <sh...@gmail.com>.
Thanks for putting together the RC, Wangda! Sorry for the late addition.
+1 (non-binding)
- Verified checksums and signatures of the artifacts
- Deployed a pseudo distributed cluster using the binary tgz, source tgz,
and git tag
- Ran basic hdfs commands
- Ran pi, sleep, distributed shell, and a sleeper service using the default
and docker runtimes
- Ran dshell using the default and docker runtimes in a secure cluster
- Verified docker runtime and privileged containers can be disabled
- Verified YARN and HDFS UI's
- Validated the change log against the commits, all discrepancies are
documented or expected
- Basic validation of Registry DNS lookups
On Thu, Apr 5, 2018 at 12:26 AM, Naganarasimha Garla <
naganarasimha_gr@apache.org> wrote:
> +1 (binding)
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Built from source
> * Deployed a pseudo Hadoop cluster
> * Verified YARN & HDFS web UIs
> * Ran sample MapReduce jobs
> * Tested NodeLabels
>
> Regards,
> + Naga
>
> On Thu, Apr 5, 2018 at 2:59 AM, Suma Shivaprasad <
> sumasai.shivaprasad@gmail.com> wrote:
>
> > +1 (non binding)
> >
> >
> > *Verified - User Group Queue mapping - Node labels with New UI- Dynamic
> > queuesThanksSuma*
> >
> >
> > On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran <st...@hortonworks.com>
> > wrote:
> >
> > > that's "dangerously interesting". I think you are right, and I also
> think
> > > it'll just be the version files which get generated
> > >
> > > anyway, +1 binding
> > >
> > > * ran my new Hadoop-3 profile on spark (SPARK-23807), with the
> committer
> > > binding, then my downstream tests. All is well, provided you also have
> a
> > > spark hive JAR patched to accept hadoop 3 as a legitimate hadoop
> version.
> > > That's an ongoing issue in the Spark project. With that JAR on my CP my
> > > downstream tests were all happy (yesterday)
> > >
> > > * today the staging files seem to be missing, at least maven is unable
> to
> > > find them even when I turn the spark snapshots-and-staging profile on.
> > > That'll be the maven dist process at play, nothing else
> > >
> > > On 4 Apr 2018, at 04:13, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > > eeleast@gmail.com>> wrote:
> > >
> > > Hi Vinod / Arpit,
> > >
> > > I checked following versions:
> > > - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
> > >
> > > Jars in maven repo [1] are *always* different from jars in the binary
> > > tarball [2]: (I only checked hadoop-yarn-api-version.jar)
> > >
> > > (Following numbers are sizes of the jar)
> > > 2.6.5:
> > > - Jar in Maven: 1896185
> > > - Jar in tarball: 1891485
> > >
> > > 2.7.5:
> > > - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> > > - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> > > (Even size is same, md5 is different)
> > >
> > > 2.8.3:
> > > - Jar in Maven: 2451433
> > > - Jar in tarball: 2438975
> > >
> > > 2.9.0:
> > > - Jar in Maven: 2791477
> > > - Jar in tarball: 2777789
> > >
> > > 3.0.1:
> > > - Jar in Maven: 2852604
> > > - Jar in tarball: 2851373
> > >
> > > I guess the differences come from our release process.
> > >
> > > Thanks,
> > > Wangda
> > >
> > > [1] Maven jars are downloaded from
> > > https://repository.apache.org/service/local/repositories/
> > > releases/content/org/apache/hadoop/hadoop-yarn-api/
> > > <version>/hadoop-yarn-api-<version>.jar
> > > [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/
> > common/
> > >
> > >
> > > On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> > > vinodkv@apache.org<ma...@apache.org>>
> > > wrote:
> > >
> > > We vote on the source code. The binaries are convenience artifacts.
> > >
> > > This is what I would do - (a) Just replace both the maven jars as well
> as
> > > the binaries to be consistent and correct. And then (b) Give a couple
> > more
> > > days for folks who tested on the binaries to reverify - I count one
> such
> > > clear vote as of now.
> > >
> > > Thanks
> > > +Vinod
> > >
> > >
> > > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > > eeleast@gmail.com>> wrote:
> > >
> > > HI Arpit,
> > >
> > > I think it won't match if we do rebuild. It should be fine as far as
> > > they're signed, correct? I don't see any policy doesn't allow this.
> > >
> > > Thanks,
> > > Wangda
> > >
> > >
> > > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <
> aagarwal@hortonworks.com<
> > > mailto:aagarwal@hortonworks.com>>
> > > wrote:
> > >
> > > Thanks Wangda, I see the shaded jars now.
> > >
> > > Are the repo jars required to be the same as the binary release? They
> > > don’t match right now, probably they got rebuilt.
> > >
> > > +1 (binding), modulo that remaining question.
> > >
> > > * Verified signatures
> > > * Verified checksums for source and binary artefacts
> > > * Sanity checked jars on r.a.o.
> > > * Built from source
> > > * Deployed to 3 node secure cluster with NameNode HA
> > > * Verified HDFS web UIs
> > > * Tried out HDFS shell commands
> > > * Ran sample MapReduce jobs
> > >
> > > Thanks!
> > >
> > >
> > > ----------------------------------------------------------------------
> > > From: Wangda Tan <wh...@gmail.com>>
> > > Date: Monday, April 2, 2018 at 9:25 PM
> > > To: Arpit Agarwal <aagarwal@hortonworks.com<mailto:
> aagarwal@hortonworks.
> > > com>>
> > > Cc: Gera Shegalov <ge...@gmail.com>>, Sunil
> G <
> > > sunilg@apache.org<ma...@apache.org>>, "
> > > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
> > > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>>,
> > Hdfs-dev <
> > > hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop
> > > Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.
> apache.org
> > > >>,
> > > "mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@
> hadoop.apache.org
> > >"
> > > <mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@
> hadoop.apache.org
> > >>,
> > > Vinod Kumar Vavilapalli <vinodkv@apache.org<mailto:vinodkv@apache.org
> >>
> > > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> > >
> > > As pointed by Arpit, the previously deployed shared jars are incorrect.
> > > Just redeployed jars and staged. @Arpit, could you please check the
> > updated
> > > Maven repo? https://repository.apache.org/content/repositories/
> > > orgapachehadoop-1092
> > >
> > > Since the jars inside binary tarballs are correct (
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> > > need roll another RC, just update Maven repo should be sufficient.
> > >
> > > Best,
> > > Wangda
> > >
> > >
> > > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com
> >
> > > wrote:
> > > Hi Arpit,
> > >
> > > Thanks for pointing out this.
> > >
> > > I just removed all .md5 files from artifacts. I found md5 checksums
> still
> > > exist in .mds files and I didn't remove them from .mds file because it
> is
> > > generated by create-release script and Apache guidance is "should not"
> > > instead of "must not". Please let me know if you think they need to be
> > > removed as well.
> > >
> > > - Wangda
> > >
> > >
> > >
> > > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> > > aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
> > > Thanks for putting together this RC, Wangda.
> > >
> > > The guidance from Apache is to omit MD5s, specifically:
> > > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> > >
> > > https://www.apache.org/dev/release-distribution#sigs-and-sums
> > >
> > >
> > >
> > >
> > > On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> > > wrote:
> > >
> > > Hi Gera,
> > >
> > > It's my bad, I thought only src/bin tarball is enough.
> > >
> > > I just uploaded all other things under artifact/ to
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> > >
> > > Please let me know if you have any other comments.
> > >
> > > Thanks,
> > > Wangda
> > >
> > >
> > > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:
> gera05@gmail.com>
> > > wrote:
> > >
> > >
> > > Thanks, Wangda!
> > >
> > > There are many more artifacts in previous votes, e.g., see
> > > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others
> the
> > > site tarball is missing.
> > >
> > > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org>
> > wrote:
> > >
> > >
> > > Thanks Wangda for initiating the release.
> > >
> > > I tested this RC built from source file.
> > >
> > >
> > > - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> > > UI.
> > > - Below feature sanity is done
> > > - Application priority
> > > - Application timeout
> > > - Intra Queue preemption with priority based
> > > - DS based affinity tests to verify placement constraints.
> > > - Tested basic NodeLabel scenarios.
> > > - Added couple of labels to few of nodes and behavior is coming
> > > correct.
> > > - Verified old UI and new YARN UI for labels.
> > > - Submitted apps to labelled cluster and it works fine.
> > > - Also performed few cli commands related to nodelabel.
> > > - Test basic HA cases and seems correct.
> > > - Tested new YARN UI . All pages are getting loaded correctly.
> > >
> > >
> > > - Sunil
> > >
> > > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com
> >
> > > wrote:
> > >
> > >
> > > Hi folks,
> > >
> > > Thanks to the many who helped with this release since Dec 2017 [1].
> > > We've
> > >
> > > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> > >
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> > >
> > > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> > >
> > > The maven artifacts are available via http://repository.apache.org<h
> > > ttp://repository.apache.org/> at
> > > https://repository.apache.org/content/repositories/
> > > orgapachehadoop-1090/
> > >
> > > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> > >
> > > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > > include the first class GPU/FPGA support on YARN, Native services,
> > > Support
> > >
> > > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > > block replicas to be provided by an external storage system, etc.
> > >
> > > For 3.1.0 RC0 vote discussion, please see [3].
> > >
> > > We’d like to use this as a starting release for 3.1.x [1], depending on
> > > how
> > >
> > > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> > > as
> > >
> > > the stable release.
> > >
> > > We have done testing with a pseudo cluster:
> > > - Ran distributed job.
> > > - GPU scheduling/isolation.
> > > - Placement constraints (intra-application anti-affinity) by using
> > > distributed shell.
> > >
> > > My +1 to start.
> > >
> > > Best,
> > > Wangda/Vinod
> > >
> > > [1]
> > >
> > > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> > > c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<
> > > http://3cyarn-dev.hadoop.apache.org/>
> > > <http://3cyarn-dev.hadoop.apache.org/>%3E
> > >
> > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.0)
> > > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> > > BY
> > >
> > > fixVersion ASC
> > > [3]
> > >
> > > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> > > 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<
> > > http://3cyarn-dev.hadoop.apache.org/>
> > > <http://3cyarn-dev.hadoop.apache.org/>%3E
> > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Shane Kumpf <sh...@gmail.com>.
Thanks for putting together the RC, Wangda! Sorry for the late addition.
+1 (non-binding)
- Verified checksums and signatures of the artifacts
- Deployed a pseudo distributed cluster using the binary tgz, source tgz,
and git tag
- Ran basic hdfs commands
- Ran pi, sleep, distributed shell, and a sleeper service using the default
and docker runtimes
- Ran dshell using the default and docker runtimes in a secure cluster
- Verified docker runtime and privileged containers can be disabled
- Verified YARN and HDFS UI's
- Validated the change log against the commits, all discrepancies are
documented or expected
- Basic validation of Registry DNS lookups
On Thu, Apr 5, 2018 at 12:26 AM, Naganarasimha Garla <
naganarasimha_gr@apache.org> wrote:
> +1 (binding)
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Built from source
> * Deployed a pseudo Hadoop cluster
> * Verified YARN & HDFS web UIs
> * Ran sample MapReduce jobs
> * Tested NodeLabels
>
> Regards,
> + Naga
>
> On Thu, Apr 5, 2018 at 2:59 AM, Suma Shivaprasad <
> sumasai.shivaprasad@gmail.com> wrote:
>
> > +1 (non binding)
> >
> >
> > *Verified - User Group Queue mapping - Node labels with New UI- Dynamic
> > queuesThanksSuma*
> >
> >
> > On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran <st...@hortonworks.com>
> > wrote:
> >
> > > that's "dangerously interesting". I think you are right, and I also
> think
> > > it'll just be the version files which get generated
> > >
> > > anyway, +1 binding
> > >
> > > * ran my new Hadoop-3 profile on spark (SPARK-23807), with the
> committer
> > > binding, then my downstream tests. All is well, provided you also have
> a
> > > spark hive JAR patched to accept hadoop 3 as a legitimate hadoop
> version.
> > > That's an ongoing issue in the Spark project. With that JAR on my CP my
> > > downstream tests were all happy (yesterday)
> > >
> > > * today the staging files seem to be missing, at least maven is unable
> to
> > > find them even when I turn the spark snapshots-and-staging profile on.
> > > That'll be the maven dist process at play, nothing else
> > >
> > > On 4 Apr 2018, at 04:13, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > > eeleast@gmail.com>> wrote:
> > >
> > > Hi Vinod / Arpit,
> > >
> > > I checked following versions:
> > > - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
> > >
> > > Jars in maven repo [1] are *always* different from jars in the binary
> > > tarball [2]: (I only checked hadoop-yarn-api-version.jar)
> > >
> > > (Following numbers are sizes of the jar)
> > > 2.6.5:
> > > - Jar in Maven: 1896185
> > > - Jar in tarball: 1891485
> > >
> > > 2.7.5:
> > > - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> > > - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> > > (Even size is same, md5 is different)
> > >
> > > 2.8.3:
> > > - Jar in Maven: 2451433
> > > - Jar in tarball: 2438975
> > >
> > > 2.9.0:
> > > - Jar in Maven: 2791477
> > > - Jar in tarball: 2777789
> > >
> > > 3.0.1:
> > > - Jar in Maven: 2852604
> > > - Jar in tarball: 2851373
> > >
> > > I guess the differences come from our release process.
> > >
> > > Thanks,
> > > Wangda
> > >
> > > [1] Maven jars are downloaded from
> > > https://repository.apache.org/service/local/repositories/
> > > releases/content/org/apache/hadoop/hadoop-yarn-api/
> > > <version>/hadoop-yarn-api-<version>.jar
> > > [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/
> > common/
> > >
> > >
> > > On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> > > vinodkv@apache.org<ma...@apache.org>>
> > > wrote:
> > >
> > > We vote on the source code. The binaries are convenience artifacts.
> > >
> > > This is what I would do - (a) Just replace both the maven jars as well
> as
> > > the binaries to be consistent and correct. And then (b) Give a couple
> > more
> > > days for folks who tested on the binaries to reverify - I count one
> such
> > > clear vote as of now.
> > >
> > > Thanks
> > > +Vinod
> > >
> > >
> > > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > > eeleast@gmail.com>> wrote:
> > >
> > > HI Arpit,
> > >
> > > I think it won't match if we do rebuild. It should be fine as far as
> > > they're signed, correct? I don't see any policy doesn't allow this.
> > >
> > > Thanks,
> > > Wangda
> > >
> > >
> > > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <
> aagarwal@hortonworks.com<
> > > mailto:aagarwal@hortonworks.com>>
> > > wrote:
> > >
> > > Thanks Wangda, I see the shaded jars now.
> > >
> > > Are the repo jars required to be the same as the binary release? They
> > > don’t match right now, probably they got rebuilt.
> > >
> > > +1 (binding), modulo that remaining question.
> > >
> > > * Verified signatures
> > > * Verified checksums for source and binary artefacts
> > > * Sanity checked jars on r.a.o.
> > > * Built from source
> > > * Deployed to 3 node secure cluster with NameNode HA
> > > * Verified HDFS web UIs
> > > * Tried out HDFS shell commands
> > > * Ran sample MapReduce jobs
> > >
> > > Thanks!
> > >
> > >
> > > ----------------------------------------------------------------------
> > > From: Wangda Tan <wh...@gmail.com>>
> > > Date: Monday, April 2, 2018 at 9:25 PM
> > > To: Arpit Agarwal <aagarwal@hortonworks.com<mailto:
> aagarwal@hortonworks.
> > > com>>
> > > Cc: Gera Shegalov <ge...@gmail.com>>, Sunil
> G <
> > > sunilg@apache.org<ma...@apache.org>>, "
> > > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
> > > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>>,
> > Hdfs-dev <
> > > hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop
> > > Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.
> apache.org
> > > >>,
> > > "mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@
> hadoop.apache.org
> > >"
> > > <mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@
> hadoop.apache.org
> > >>,
> > > Vinod Kumar Vavilapalli <vinodkv@apache.org<mailto:vinodkv@apache.org
> >>
> > > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> > >
> > > As pointed by Arpit, the previously deployed shared jars are incorrect.
> > > Just redeployed jars and staged. @Arpit, could you please check the
> > updated
> > > Maven repo? https://repository.apache.org/content/repositories/
> > > orgapachehadoop-1092
> > >
> > > Since the jars inside binary tarballs are correct (
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> > > need roll another RC, just update Maven repo should be sufficient.
> > >
> > > Best,
> > > Wangda
> > >
> > >
> > > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com
> >
> > > wrote:
> > > Hi Arpit,
> > >
> > > Thanks for pointing out this.
> > >
> > > I just removed all .md5 files from artifacts. I found md5 checksums
> still
> > > exist in .mds files and I didn't remove them from .mds file because it
> is
> > > generated by create-release script and Apache guidance is "should not"
> > > instead of "must not". Please let me know if you think they need to be
> > > removed as well.
> > >
> > > - Wangda
> > >
> > >
> > >
> > > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> > > aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
> > > Thanks for putting together this RC, Wangda.
> > >
> > > The guidance from Apache is to omit MD5s, specifically:
> > > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> > >
> > > https://www.apache.org/dev/release-distribution#sigs-and-sums
> > >
> > >
> > >
> > >
> > > On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> > > wrote:
> > >
> > > Hi Gera,
> > >
> > > It's my bad, I thought only src/bin tarball is enough.
> > >
> > > I just uploaded all other things under artifact/ to
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> > >
> > > Please let me know if you have any other comments.
> > >
> > > Thanks,
> > > Wangda
> > >
> > >
> > > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:
> gera05@gmail.com>
> > > wrote:
> > >
> > >
> > > Thanks, Wangda!
> > >
> > > There are many more artifacts in previous votes, e.g., see
> > > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others
> the
> > > site tarball is missing.
> > >
> > > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org>
> > wrote:
> > >
> > >
> > > Thanks Wangda for initiating the release.
> > >
> > > I tested this RC built from source file.
> > >
> > >
> > > - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> > > UI.
> > > - Below feature sanity is done
> > > - Application priority
> > > - Application timeout
> > > - Intra Queue preemption with priority based
> > > - DS based affinity tests to verify placement constraints.
> > > - Tested basic NodeLabel scenarios.
> > > - Added couple of labels to few of nodes and behavior is coming
> > > correct.
> > > - Verified old UI and new YARN UI for labels.
> > > - Submitted apps to labelled cluster and it works fine.
> > > - Also performed few cli commands related to nodelabel.
> > > - Test basic HA cases and seems correct.
> > > - Tested new YARN UI . All pages are getting loaded correctly.
> > >
> > >
> > > - Sunil
> > >
> > > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com
> >
> > > wrote:
> > >
> > >
> > > Hi folks,
> > >
> > > Thanks to the many who helped with this release since Dec 2017 [1].
> > > We've
> > >
> > > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> > >
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> > >
> > > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> > >
> > > The maven artifacts are available via http://repository.apache.org<h
> > > ttp://repository.apache.org/> at
> > > https://repository.apache.org/content/repositories/
> > > orgapachehadoop-1090/
> > >
> > > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> > >
> > > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > > include the first class GPU/FPGA support on YARN, Native services,
> > > Support
> > >
> > > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > > block replicas to be provided by an external storage system, etc.
> > >
> > > For 3.1.0 RC0 vote discussion, please see [3].
> > >
> > > We’d like to use this as a starting release for 3.1.x [1], depending on
> > > how
> > >
> > > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> > > as
> > >
> > > the stable release.
> > >
> > > We have done testing with a pseudo cluster:
> > > - Ran distributed job.
> > > - GPU scheduling/isolation.
> > > - Placement constraints (intra-application anti-affinity) by using
> > > distributed shell.
> > >
> > > My +1 to start.
> > >
> > > Best,
> > > Wangda/Vinod
> > >
> > > [1]
> > >
> > > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> > > c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<
> > > http://3cyarn-dev.hadoop.apache.org/>
> > > <http://3cyarn-dev.hadoop.apache.org/>%3E
> > >
> > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.0)
> > > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> > > BY
> > >
> > > fixVersion ASC
> > > [3]
> > >
> > > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> > > 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<
> > > http://3cyarn-dev.hadoop.apache.org/>
> > > <http://3cyarn-dev.hadoop.apache.org/>%3E
> > >
> > >
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Naganarasimha Garla <na...@apache.org>.
+1 (binding)
* Verified signatures
* Verified checksums for source and binary artefacts
* Built from source
* Deployed a pseudo Hadoop cluster
* Verified YARN & HDFS web UIs
* Ran sample MapReduce jobs
* Tested NodeLabels
Regards,
+ Naga
On Thu, Apr 5, 2018 at 2:59 AM, Suma Shivaprasad <
sumasai.shivaprasad@gmail.com> wrote:
> +1 (non binding)
>
>
> *Verified - User Group Queue mapping - Node labels with New UI- Dynamic
> queuesThanksSuma*
>
>
> On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
> > that's "dangerously interesting". I think you are right, and I also think
> > it'll just be the version files which get generated
> >
> > anyway, +1 binding
> >
> > * ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer
> > binding, then my downstream tests. All is well, provided you also have a
> > spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version.
> > That's an ongoing issue in the Spark project. With that JAR on my CP my
> > downstream tests were all happy (yesterday)
> >
> > * today the staging files seem to be missing, at least maven is unable to
> > find them even when I turn the spark snapshots-and-staging profile on.
> > That'll be the maven dist process at play, nothing else
> >
> > On 4 Apr 2018, at 04:13, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > eeleast@gmail.com>> wrote:
> >
> > Hi Vinod / Arpit,
> >
> > I checked following versions:
> > - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
> >
> > Jars in maven repo [1] are *always* different from jars in the binary
> > tarball [2]: (I only checked hadoop-yarn-api-version.jar)
> >
> > (Following numbers are sizes of the jar)
> > 2.6.5:
> > - Jar in Maven: 1896185
> > - Jar in tarball: 1891485
> >
> > 2.7.5:
> > - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> > - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> > (Even size is same, md5 is different)
> >
> > 2.8.3:
> > - Jar in Maven: 2451433
> > - Jar in tarball: 2438975
> >
> > 2.9.0:
> > - Jar in Maven: 2791477
> > - Jar in tarball: 2777789
> >
> > 3.0.1:
> > - Jar in Maven: 2852604
> > - Jar in tarball: 2851373
> >
> > I guess the differences come from our release process.
> >
> > Thanks,
> > Wangda
> >
> > [1] Maven jars are downloaded from
> > https://repository.apache.org/service/local/repositories/
> > releases/content/org/apache/hadoop/hadoop-yarn-api/
> > <version>/hadoop-yarn-api-<version>.jar
> > [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/
> common/
> >
> >
> > On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> > vinodkv@apache.org<ma...@apache.org>>
> > wrote:
> >
> > We vote on the source code. The binaries are convenience artifacts.
> >
> > This is what I would do - (a) Just replace both the maven jars as well as
> > the binaries to be consistent and correct. And then (b) Give a couple
> more
> > days for folks who tested on the binaries to reverify - I count one such
> > clear vote as of now.
> >
> > Thanks
> > +Vinod
> >
> >
> > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > eeleast@gmail.com>> wrote:
> >
> > HI Arpit,
> >
> > I think it won't match if we do rebuild. It should be fine as far as
> > they're signed, correct? I don't see any policy doesn't allow this.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com<
> > mailto:aagarwal@hortonworks.com>>
> > wrote:
> >
> > Thanks Wangda, I see the shaded jars now.
> >
> > Are the repo jars required to be the same as the binary release? They
> > don’t match right now, probably they got rebuilt.
> >
> > +1 (binding), modulo that remaining question.
> >
> > * Verified signatures
> > * Verified checksums for source and binary artefacts
> > * Sanity checked jars on r.a.o.
> > * Built from source
> > * Deployed to 3 node secure cluster with NameNode HA
> > * Verified HDFS web UIs
> > * Tried out HDFS shell commands
> > * Ran sample MapReduce jobs
> >
> > Thanks!
> >
> >
> > ----------------------------------------------------------------------
> > From: Wangda Tan <wh...@gmail.com>>
> > Date: Monday, April 2, 2018 at 9:25 PM
> > To: Arpit Agarwal <aagarwal@hortonworks.com<mailto:aagarwal@hortonworks.
> > com>>
> > Cc: Gera Shegalov <ge...@gmail.com>>, Sunil G <
> > sunilg@apache.org<ma...@apache.org>>, "
> > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
> > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>>,
> Hdfs-dev <
> > hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop
> > Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org
> > >>,
> > "mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org
> >"
> > <mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org
> >>,
> > Vinod Kumar Vavilapalli <vi...@apache.org>>
> > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> >
> > As pointed by Arpit, the previously deployed shared jars are incorrect.
> > Just redeployed jars and staged. @Arpit, could you please check the
> updated
> > Maven repo? https://repository.apache.org/content/repositories/
> > orgapachehadoop-1092
> >
> > Since the jars inside binary tarballs are correct (
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> > need roll another RC, just update Maven repo should be sufficient.
> >
> > Best,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> > wrote:
> > Hi Arpit,
> >
> > Thanks for pointing out this.
> >
> > I just removed all .md5 files from artifacts. I found md5 checksums still
> > exist in .mds files and I didn't remove them from .mds file because it is
> > generated by create-release script and Apache guidance is "should not"
> > instead of "must not". Please let me know if you think they need to be
> > removed as well.
> >
> > - Wangda
> >
> >
> >
> > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> > aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
> > Thanks for putting together this RC, Wangda.
> >
> > The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> >
> > https://www.apache.org/dev/release-distribution#sigs-and-sums
> >
> >
> >
> >
> > On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> > wrote:
> >
> > Hi Gera,
> >
> > It's my bad, I thought only src/bin tarball is enough.
> >
> > I just uploaded all other things under artifact/ to
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> >
> > Please let me know if you have any other comments.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> > wrote:
> >
> >
> > Thanks, Wangda!
> >
> > There are many more artifacts in previous votes, e.g., see
> > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> > site tarball is missing.
> >
> > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org>
> wrote:
> >
> >
> > Thanks Wangda for initiating the release.
> >
> > I tested this RC built from source file.
> >
> >
> > - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> > UI.
> > - Below feature sanity is done
> > - Application priority
> > - Application timeout
> > - Intra Queue preemption with priority based
> > - DS based affinity tests to verify placement constraints.
> > - Tested basic NodeLabel scenarios.
> > - Added couple of labels to few of nodes and behavior is coming
> > correct.
> > - Verified old UI and new YARN UI for labels.
> > - Submitted apps to labelled cluster and it works fine.
> > - Also performed few cli commands related to nodelabel.
> > - Test basic HA cases and seems correct.
> > - Tested new YARN UI . All pages are getting loaded correctly.
> >
> >
> > - Sunil
> >
> > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> > wrote:
> >
> >
> > Hi folks,
> >
> > Thanks to the many who helped with this release since Dec 2017 [1].
> > We've
> >
> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >
> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >
> > The maven artifacts are available via http://repository.apache.org<h
> > ttp://repository.apache.org/> at
> > https://repository.apache.org/content/repositories/
> > orgapachehadoop-1090/
> >
> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >
> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > include the first class GPU/FPGA support on YARN, Native services,
> > Support
> >
> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > block replicas to be provided by an external storage system, etc.
> >
> > For 3.1.0 RC0 vote discussion, please see [3].
> >
> > We’d like to use this as a starting release for 3.1.x [1], depending on
> > how
> >
> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> > as
> >
> > the stable release.
> >
> > We have done testing with a pseudo cluster:
> > - Ran distributed job.
> > - GPU scheduling/isolation.
> > - Placement constraints (intra-application anti-affinity) by using
> > distributed shell.
> >
> > My +1 to start.
> >
> > Best,
> > Wangda/Vinod
> >
> > [1]
> >
> > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> > c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<
> > http://3cyarn-dev.hadoop.apache.org/>
> > <http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> > BY
> >
> > fixVersion ASC
> > [3]
> >
> > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> > 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<
> > http://3cyarn-dev.hadoop.apache.org/>
> > <http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Naganarasimha Garla <na...@apache.org>.
+1 (binding)
* Verified signatures
* Verified checksums for source and binary artefacts
* Built from source
* Deployed a pseudo Hadoop cluster
* Verified YARN & HDFS web UIs
* Ran sample MapReduce jobs
* Tested NodeLabels
Regards,
+ Naga
On Thu, Apr 5, 2018 at 2:59 AM, Suma Shivaprasad <
sumasai.shivaprasad@gmail.com> wrote:
> +1 (non binding)
>
>
> *Verified - User Group Queue mapping - Node labels with New UI- Dynamic
> queuesThanksSuma*
>
>
> On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
> > that's "dangerously interesting". I think you are right, and I also think
> > it'll just be the version files which get generated
> >
> > anyway, +1 binding
> >
> > * ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer
> > binding, then my downstream tests. All is well, provided you also have a
> > spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version.
> > That's an ongoing issue in the Spark project. With that JAR on my CP my
> > downstream tests were all happy (yesterday)
> >
> > * today the staging files seem to be missing, at least maven is unable to
> > find them even when I turn the spark snapshots-and-staging profile on.
> > That'll be the maven dist process at play, nothing else
> >
> > On 4 Apr 2018, at 04:13, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > eeleast@gmail.com>> wrote:
> >
> > Hi Vinod / Arpit,
> >
> > I checked following versions:
> > - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
> >
> > Jars in maven repo [1] are *always* different from jars in the binary
> > tarball [2]: (I only checked hadoop-yarn-api-version.jar)
> >
> > (Following numbers are sizes of the jar)
> > 2.6.5:
> > - Jar in Maven: 1896185
> > - Jar in tarball: 1891485
> >
> > 2.7.5:
> > - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> > - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> > (Even size is same, md5 is different)
> >
> > 2.8.3:
> > - Jar in Maven: 2451433
> > - Jar in tarball: 2438975
> >
> > 2.9.0:
> > - Jar in Maven: 2791477
> > - Jar in tarball: 2777789
> >
> > 3.0.1:
> > - Jar in Maven: 2852604
> > - Jar in tarball: 2851373
> >
> > I guess the differences come from our release process.
> >
> > Thanks,
> > Wangda
> >
> > [1] Maven jars are downloaded from
> > https://repository.apache.org/service/local/repositories/
> > releases/content/org/apache/hadoop/hadoop-yarn-api/
> > <version>/hadoop-yarn-api-<version>.jar
> > [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/
> common/
> >
> >
> > On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> > vinodkv@apache.org<ma...@apache.org>>
> > wrote:
> >
> > We vote on the source code. The binaries are convenience artifacts.
> >
> > This is what I would do - (a) Just replace both the maven jars as well as
> > the binaries to be consistent and correct. And then (b) Give a couple
> more
> > days for folks who tested on the binaries to reverify - I count one such
> > clear vote as of now.
> >
> > Thanks
> > +Vinod
> >
> >
> > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > eeleast@gmail.com>> wrote:
> >
> > HI Arpit,
> >
> > I think it won't match if we do rebuild. It should be fine as far as
> > they're signed, correct? I don't see any policy doesn't allow this.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com<
> > mailto:aagarwal@hortonworks.com>>
> > wrote:
> >
> > Thanks Wangda, I see the shaded jars now.
> >
> > Are the repo jars required to be the same as the binary release? They
> > don’t match right now, probably they got rebuilt.
> >
> > +1 (binding), modulo that remaining question.
> >
> > * Verified signatures
> > * Verified checksums for source and binary artefacts
> > * Sanity checked jars on r.a.o.
> > * Built from source
> > * Deployed to 3 node secure cluster with NameNode HA
> > * Verified HDFS web UIs
> > * Tried out HDFS shell commands
> > * Ran sample MapReduce jobs
> >
> > Thanks!
> >
> >
> > ----------------------------------------------------------------------
> > From: Wangda Tan <wh...@gmail.com>>
> > Date: Monday, April 2, 2018 at 9:25 PM
> > To: Arpit Agarwal <aagarwal@hortonworks.com<mailto:aagarwal@hortonworks.
> > com>>
> > Cc: Gera Shegalov <ge...@gmail.com>>, Sunil G <
> > sunilg@apache.org<ma...@apache.org>>, "
> > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
> > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>>,
> Hdfs-dev <
> > hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop
> > Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org
> > >>,
> > "mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org
> >"
> > <mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org
> >>,
> > Vinod Kumar Vavilapalli <vi...@apache.org>>
> > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> >
> > As pointed by Arpit, the previously deployed shared jars are incorrect.
> > Just redeployed jars and staged. @Arpit, could you please check the
> updated
> > Maven repo? https://repository.apache.org/content/repositories/
> > orgapachehadoop-1092
> >
> > Since the jars inside binary tarballs are correct (
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> > need roll another RC, just update Maven repo should be sufficient.
> >
> > Best,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> > wrote:
> > Hi Arpit,
> >
> > Thanks for pointing out this.
> >
> > I just removed all .md5 files from artifacts. I found md5 checksums still
> > exist in .mds files and I didn't remove them from .mds file because it is
> > generated by create-release script and Apache guidance is "should not"
> > instead of "must not". Please let me know if you think they need to be
> > removed as well.
> >
> > - Wangda
> >
> >
> >
> > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> > aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
> > Thanks for putting together this RC, Wangda.
> >
> > The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> >
> > https://www.apache.org/dev/release-distribution#sigs-and-sums
> >
> >
> >
> >
> > On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> > wrote:
> >
> > Hi Gera,
> >
> > It's my bad, I thought only src/bin tarball is enough.
> >
> > I just uploaded all other things under artifact/ to
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> >
> > Please let me know if you have any other comments.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> > wrote:
> >
> >
> > Thanks, Wangda!
> >
> > There are many more artifacts in previous votes, e.g., see
> > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> > site tarball is missing.
> >
> > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org>
> wrote:
> >
> >
> > Thanks Wangda for initiating the release.
> >
> > I tested this RC built from source file.
> >
> >
> > - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> > UI.
> > - Below feature sanity is done
> > - Application priority
> > - Application timeout
> > - Intra Queue preemption with priority based
> > - DS based affinity tests to verify placement constraints.
> > - Tested basic NodeLabel scenarios.
> > - Added couple of labels to few of nodes and behavior is coming
> > correct.
> > - Verified old UI and new YARN UI for labels.
> > - Submitted apps to labelled cluster and it works fine.
> > - Also performed few cli commands related to nodelabel.
> > - Test basic HA cases and seems correct.
> > - Tested new YARN UI . All pages are getting loaded correctly.
> >
> >
> > - Sunil
> >
> > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> > wrote:
> >
> >
> > Hi folks,
> >
> > Thanks to the many who helped with this release since Dec 2017 [1].
> > We've
> >
> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >
> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >
> > The maven artifacts are available via http://repository.apache.org<h
> > ttp://repository.apache.org/> at
> > https://repository.apache.org/content/repositories/
> > orgapachehadoop-1090/
> >
> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >
> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > include the first class GPU/FPGA support on YARN, Native services,
> > Support
> >
> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > block replicas to be provided by an external storage system, etc.
> >
> > For 3.1.0 RC0 vote discussion, please see [3].
> >
> > We’d like to use this as a starting release for 3.1.x [1], depending on
> > how
> >
> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> > as
> >
> > the stable release.
> >
> > We have done testing with a pseudo cluster:
> > - Ran distributed job.
> > - GPU scheduling/isolation.
> > - Placement constraints (intra-application anti-affinity) by using
> > distributed shell.
> >
> > My +1 to start.
> >
> > Best,
> > Wangda/Vinod
> >
> > [1]
> >
> > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> > c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<
> > http://3cyarn-dev.hadoop.apache.org/>
> > <http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> > BY
> >
> > fixVersion ASC
> > [3]
> >
> > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> > 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<
> > http://3cyarn-dev.hadoop.apache.org/>
> > <http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Naganarasimha Garla <na...@apache.org>.
+1 (binding)
* Verified signatures
* Verified checksums for source and binary artefacts
* Built from source
* Deployed a pseudo Hadoop cluster
* Verified YARN & HDFS web UIs
* Ran sample MapReduce jobs
* Tested NodeLabels
Regards,
+ Naga
On Thu, Apr 5, 2018 at 2:59 AM, Suma Shivaprasad <
sumasai.shivaprasad@gmail.com> wrote:
> +1 (non binding)
>
>
> *Verified - User Group Queue mapping - Node labels with New UI- Dynamic
> queuesThanksSuma*
>
>
> On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
> > that's "dangerously interesting". I think you are right, and I also think
> > it'll just be the version files which get generated
> >
> > anyway, +1 binding
> >
> > * ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer
> > binding, then my downstream tests. All is well, provided you also have a
> > spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version.
> > That's an ongoing issue in the Spark project. With that JAR on my CP my
> > downstream tests were all happy (yesterday)
> >
> > * today the staging files seem to be missing, at least maven is unable to
> > find them even when I turn the spark snapshots-and-staging profile on.
> > That'll be the maven dist process at play, nothing else
> >
> > On 4 Apr 2018, at 04:13, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > eeleast@gmail.com>> wrote:
> >
> > Hi Vinod / Arpit,
> >
> > I checked following versions:
> > - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
> >
> > Jars in maven repo [1] are *always* different from jars in the binary
> > tarball [2]: (I only checked hadoop-yarn-api-version.jar)
> >
> > (Following numbers are sizes of the jar)
> > 2.6.5:
> > - Jar in Maven: 1896185
> > - Jar in tarball: 1891485
> >
> > 2.7.5:
> > - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> > - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> > (Even size is same, md5 is different)
> >
> > 2.8.3:
> > - Jar in Maven: 2451433
> > - Jar in tarball: 2438975
> >
> > 2.9.0:
> > - Jar in Maven: 2791477
> > - Jar in tarball: 2777789
> >
> > 3.0.1:
> > - Jar in Maven: 2852604
> > - Jar in tarball: 2851373
> >
> > I guess the differences come from our release process.
> >
> > Thanks,
> > Wangda
> >
> > [1] Maven jars are downloaded from
> > https://repository.apache.org/service/local/repositories/
> > releases/content/org/apache/hadoop/hadoop-yarn-api/
> > <version>/hadoop-yarn-api-<version>.jar
> > [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/
> common/
> >
> >
> > On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> > vinodkv@apache.org<ma...@apache.org>>
> > wrote:
> >
> > We vote on the source code. The binaries are convenience artifacts.
> >
> > This is what I would do - (a) Just replace both the maven jars as well as
> > the binaries to be consistent and correct. And then (b) Give a couple
> more
> > days for folks who tested on the binaries to reverify - I count one such
> > clear vote as of now.
> >
> > Thanks
> > +Vinod
> >
> >
> > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > eeleast@gmail.com>> wrote:
> >
> > HI Arpit,
> >
> > I think it won't match if we do rebuild. It should be fine as far as
> > they're signed, correct? I don't see any policy doesn't allow this.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com<
> > mailto:aagarwal@hortonworks.com>>
> > wrote:
> >
> > Thanks Wangda, I see the shaded jars now.
> >
> > Are the repo jars required to be the same as the binary release? They
> > don’t match right now, probably they got rebuilt.
> >
> > +1 (binding), modulo that remaining question.
> >
> > * Verified signatures
> > * Verified checksums for source and binary artefacts
> > * Sanity checked jars on r.a.o.
> > * Built from source
> > * Deployed to 3 node secure cluster with NameNode HA
> > * Verified HDFS web UIs
> > * Tried out HDFS shell commands
> > * Ran sample MapReduce jobs
> >
> > Thanks!
> >
> >
> > ----------------------------------------------------------------------
> > From: Wangda Tan <wh...@gmail.com>>
> > Date: Monday, April 2, 2018 at 9:25 PM
> > To: Arpit Agarwal <aagarwal@hortonworks.com<mailto:aagarwal@hortonworks.
> > com>>
> > Cc: Gera Shegalov <ge...@gmail.com>>, Sunil G <
> > sunilg@apache.org<ma...@apache.org>>, "
> > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
> > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>>,
> Hdfs-dev <
> > hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop
> > Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org
> > >>,
> > "mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org
> >"
> > <mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org
> >>,
> > Vinod Kumar Vavilapalli <vi...@apache.org>>
> > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> >
> > As pointed by Arpit, the previously deployed shared jars are incorrect.
> > Just redeployed jars and staged. @Arpit, could you please check the
> updated
> > Maven repo? https://repository.apache.org/content/repositories/
> > orgapachehadoop-1092
> >
> > Since the jars inside binary tarballs are correct (
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> > need roll another RC, just update Maven repo should be sufficient.
> >
> > Best,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> > wrote:
> > Hi Arpit,
> >
> > Thanks for pointing out this.
> >
> > I just removed all .md5 files from artifacts. I found md5 checksums still
> > exist in .mds files and I didn't remove them from .mds file because it is
> > generated by create-release script and Apache guidance is "should not"
> > instead of "must not". Please let me know if you think they need to be
> > removed as well.
> >
> > - Wangda
> >
> >
> >
> > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> > aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
> > Thanks for putting together this RC, Wangda.
> >
> > The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> >
> > https://www.apache.org/dev/release-distribution#sigs-and-sums
> >
> >
> >
> >
> > On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> > wrote:
> >
> > Hi Gera,
> >
> > It's my bad, I thought only src/bin tarball is enough.
> >
> > I just uploaded all other things under artifact/ to
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> >
> > Please let me know if you have any other comments.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> > wrote:
> >
> >
> > Thanks, Wangda!
> >
> > There are many more artifacts in previous votes, e.g., see
> > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> > site tarball is missing.
> >
> > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org>
> wrote:
> >
> >
> > Thanks Wangda for initiating the release.
> >
> > I tested this RC built from source file.
> >
> >
> > - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> > UI.
> > - Below feature sanity is done
> > - Application priority
> > - Application timeout
> > - Intra Queue preemption with priority based
> > - DS based affinity tests to verify placement constraints.
> > - Tested basic NodeLabel scenarios.
> > - Added couple of labels to few of nodes and behavior is coming
> > correct.
> > - Verified old UI and new YARN UI for labels.
> > - Submitted apps to labelled cluster and it works fine.
> > - Also performed few cli commands related to nodelabel.
> > - Test basic HA cases and seems correct.
> > - Tested new YARN UI . All pages are getting loaded correctly.
> >
> >
> > - Sunil
> >
> > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> > wrote:
> >
> >
> > Hi folks,
> >
> > Thanks to the many who helped with this release since Dec 2017 [1].
> > We've
> >
> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >
> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >
> > The maven artifacts are available via http://repository.apache.org<h
> > ttp://repository.apache.org/> at
> > https://repository.apache.org/content/repositories/
> > orgapachehadoop-1090/
> >
> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >
> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > include the first class GPU/FPGA support on YARN, Native services,
> > Support
> >
> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > block replicas to be provided by an external storage system, etc.
> >
> > For 3.1.0 RC0 vote discussion, please see [3].
> >
> > We’d like to use this as a starting release for 3.1.x [1], depending on
> > how
> >
> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> > as
> >
> > the stable release.
> >
> > We have done testing with a pseudo cluster:
> > - Ran distributed job.
> > - GPU scheduling/isolation.
> > - Placement constraints (intra-application anti-affinity) by using
> > distributed shell.
> >
> > My +1 to start.
> >
> > Best,
> > Wangda/Vinod
> >
> > [1]
> >
> > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> > c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<
> > http://3cyarn-dev.hadoop.apache.org/>
> > <http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> > BY
> >
> > fixVersion ASC
> > [3]
> >
> > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> > 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<
> > http://3cyarn-dev.hadoop.apache.org/>
> > <http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Naganarasimha Garla <na...@apache.org>.
+1 (binding)
* Verified signatures
* Verified checksums for source and binary artefacts
* Built from source
* Deployed a pseudo Hadoop cluster
* Verified YARN & HDFS web UIs
* Ran sample MapReduce jobs
* Tested NodeLabels
Regards,
+ Naga
On Thu, Apr 5, 2018 at 2:59 AM, Suma Shivaprasad <
sumasai.shivaprasad@gmail.com> wrote:
> +1 (non binding)
>
>
> *Verified - User Group Queue mapping - Node labels with New UI- Dynamic
> queuesThanksSuma*
>
>
> On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
> > that's "dangerously interesting". I think you are right, and I also think
> > it'll just be the version files which get generated
> >
> > anyway, +1 binding
> >
> > * ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer
> > binding, then my downstream tests. All is well, provided you also have a
> > spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version.
> > That's an ongoing issue in the Spark project. With that JAR on my CP my
> > downstream tests were all happy (yesterday)
> >
> > * today the staging files seem to be missing, at least maven is unable to
> > find them even when I turn the spark snapshots-and-staging profile on.
> > That'll be the maven dist process at play, nothing else
> >
> > On 4 Apr 2018, at 04:13, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > eeleast@gmail.com>> wrote:
> >
> > Hi Vinod / Arpit,
> >
> > I checked following versions:
> > - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
> >
> > Jars in maven repo [1] are *always* different from jars in the binary
> > tarball [2]: (I only checked hadoop-yarn-api-version.jar)
> >
> > (Following numbers are sizes of the jar)
> > 2.6.5:
> > - Jar in Maven: 1896185
> > - Jar in tarball: 1891485
> >
> > 2.7.5:
> > - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> > - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> > (Even size is same, md5 is different)
> >
> > 2.8.3:
> > - Jar in Maven: 2451433
> > - Jar in tarball: 2438975
> >
> > 2.9.0:
> > - Jar in Maven: 2791477
> > - Jar in tarball: 2777789
> >
> > 3.0.1:
> > - Jar in Maven: 2852604
> > - Jar in tarball: 2851373
> >
> > I guess the differences come from our release process.
> >
> > Thanks,
> > Wangda
> >
> > [1] Maven jars are downloaded from
> > https://repository.apache.org/service/local/repositories/
> > releases/content/org/apache/hadoop/hadoop-yarn-api/
> > <version>/hadoop-yarn-api-<version>.jar
> > [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/
> common/
> >
> >
> > On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> > vinodkv@apache.org<ma...@apache.org>>
> > wrote:
> >
> > We vote on the source code. The binaries are convenience artifacts.
> >
> > This is what I would do - (a) Just replace both the maven jars as well as
> > the binaries to be consistent and correct. And then (b) Give a couple
> more
> > days for folks who tested on the binaries to reverify - I count one such
> > clear vote as of now.
> >
> > Thanks
> > +Vinod
> >
> >
> > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > eeleast@gmail.com>> wrote:
> >
> > HI Arpit,
> >
> > I think it won't match if we do rebuild. It should be fine as far as
> > they're signed, correct? I don't see any policy doesn't allow this.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com<
> > mailto:aagarwal@hortonworks.com>>
> > wrote:
> >
> > Thanks Wangda, I see the shaded jars now.
> >
> > Are the repo jars required to be the same as the binary release? They
> > don’t match right now, probably they got rebuilt.
> >
> > +1 (binding), modulo that remaining question.
> >
> > * Verified signatures
> > * Verified checksums for source and binary artefacts
> > * Sanity checked jars on r.a.o.
> > * Built from source
> > * Deployed to 3 node secure cluster with NameNode HA
> > * Verified HDFS web UIs
> > * Tried out HDFS shell commands
> > * Ran sample MapReduce jobs
> >
> > Thanks!
> >
> >
> > ----------------------------------------------------------------------
> > From: Wangda Tan <wh...@gmail.com>>
> > Date: Monday, April 2, 2018 at 9:25 PM
> > To: Arpit Agarwal <aagarwal@hortonworks.com<mailto:aagarwal@hortonworks.
> > com>>
> > Cc: Gera Shegalov <ge...@gmail.com>>, Sunil G <
> > sunilg@apache.org<ma...@apache.org>>, "
> > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
> > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>>,
> Hdfs-dev <
> > hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop
> > Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org
> > >>,
> > "mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org
> >"
> > <mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org
> >>,
> > Vinod Kumar Vavilapalli <vi...@apache.org>>
> > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> >
> > As pointed by Arpit, the previously deployed shared jars are incorrect.
> > Just redeployed jars and staged. @Arpit, could you please check the
> updated
> > Maven repo? https://repository.apache.org/content/repositories/
> > orgapachehadoop-1092
> >
> > Since the jars inside binary tarballs are correct (
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> > need roll another RC, just update Maven repo should be sufficient.
> >
> > Best,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> > wrote:
> > Hi Arpit,
> >
> > Thanks for pointing out this.
> >
> > I just removed all .md5 files from artifacts. I found md5 checksums still
> > exist in .mds files and I didn't remove them from .mds file because it is
> > generated by create-release script and Apache guidance is "should not"
> > instead of "must not". Please let me know if you think they need to be
> > removed as well.
> >
> > - Wangda
> >
> >
> >
> > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> > aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
> > Thanks for putting together this RC, Wangda.
> >
> > The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> >
> > https://www.apache.org/dev/release-distribution#sigs-and-sums
> >
> >
> >
> >
> > On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> > wrote:
> >
> > Hi Gera,
> >
> > It's my bad, I thought only src/bin tarball is enough.
> >
> > I just uploaded all other things under artifact/ to
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> >
> > Please let me know if you have any other comments.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> > wrote:
> >
> >
> > Thanks, Wangda!
> >
> > There are many more artifacts in previous votes, e.g., see
> > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> > site tarball is missing.
> >
> > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org>
> wrote:
> >
> >
> > Thanks Wangda for initiating the release.
> >
> > I tested this RC built from source file.
> >
> >
> > - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> > UI.
> > - Below feature sanity is done
> > - Application priority
> > - Application timeout
> > - Intra Queue preemption with priority based
> > - DS based affinity tests to verify placement constraints.
> > - Tested basic NodeLabel scenarios.
> > - Added couple of labels to few of nodes and behavior is coming
> > correct.
> > - Verified old UI and new YARN UI for labels.
> > - Submitted apps to labelled cluster and it works fine.
> > - Also performed few cli commands related to nodelabel.
> > - Test basic HA cases and seems correct.
> > - Tested new YARN UI . All pages are getting loaded correctly.
> >
> >
> > - Sunil
> >
> > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> > wrote:
> >
> >
> > Hi folks,
> >
> > Thanks to the many who helped with this release since Dec 2017 [1].
> > We've
> >
> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >
> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >
> > The maven artifacts are available via http://repository.apache.org<h
> > ttp://repository.apache.org/> at
> > https://repository.apache.org/content/repositories/
> > orgapachehadoop-1090/
> >
> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >
> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > include the first class GPU/FPGA support on YARN, Native services,
> > Support
> >
> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > block replicas to be provided by an external storage system, etc.
> >
> > For 3.1.0 RC0 vote discussion, please see [3].
> >
> > We’d like to use this as a starting release for 3.1.x [1], depending on
> > how
> >
> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> > as
> >
> > the stable release.
> >
> > We have done testing with a pseudo cluster:
> > - Ran distributed job.
> > - GPU scheduling/isolation.
> > - Placement constraints (intra-application anti-affinity) by using
> > distributed shell.
> >
> > My +1 to start.
> >
> > Best,
> > Wangda/Vinod
> >
> > [1]
> >
> > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> > c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<
> > http://3cyarn-dev.hadoop.apache.org/>
> > <http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> > BY
> >
> > fixVersion ASC
> > [3]
> >
> > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> > 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<
> > http://3cyarn-dev.hadoop.apache.org/>
> > <http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Suma Shivaprasad <su...@gmail.com>.
+1 (non binding)
*Verified - User Group Queue mapping - Node labels with New UI- Dynamic
queuesThanksSuma*
On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran <st...@hortonworks.com>
wrote:
> that's "dangerously interesting". I think you are right, and I also think
> it'll just be the version files which get generated
>
> anyway, +1 binding
>
> * ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer
> binding, then my downstream tests. All is well, provided you also have a
> spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version.
> That's an ongoing issue in the Spark project. With that JAR on my CP my
> downstream tests were all happy (yesterday)
>
> * today the staging files seem to be missing, at least maven is unable to
> find them even when I turn the spark snapshots-and-staging profile on.
> That'll be the maven dist process at play, nothing else
>
> On 4 Apr 2018, at 04:13, Wangda Tan <wheeleast@gmail.com<mailto:wh
> eeleast@gmail.com>> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org<ma...@apache.org>>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com<mailto:wh
> eeleast@gmail.com>> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com<
> mailto:aagarwal@hortonworks.com>>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagarwal@hortonworks.com<mailto:aagarwal@hortonworks.
> com>>
> Cc: Gera Shegalov <ge...@gmail.com>>, Sunil G <
> sunilg@apache.org<ma...@apache.org>>, "
> yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
> yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop
> Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org
> >>,
> "mapreduce-dev@hadoop.apache.org<ma...@hadoop.apache.org>"
> <ma...@hadoop.apache.org>>,
> Vinod Kumar Vavilapalli <vi...@apache.org>>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org<h
> ttp://repository.apache.org/> at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<
> http://3cyarn-dev.hadoop.apache.org/>
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<
> http://3cyarn-dev.hadoop.apache.org/>
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Suma Shivaprasad <su...@gmail.com>.
+1 (non binding)
*Verified - User Group Queue mapping - Node labels with New UI- Dynamic
queuesThanksSuma*
On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran <st...@hortonworks.com>
wrote:
> that's "dangerously interesting". I think you are right, and I also think
> it'll just be the version files which get generated
>
> anyway, +1 binding
>
> * ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer
> binding, then my downstream tests. All is well, provided you also have a
> spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version.
> That's an ongoing issue in the Spark project. With that JAR on my CP my
> downstream tests were all happy (yesterday)
>
> * today the staging files seem to be missing, at least maven is unable to
> find them even when I turn the spark snapshots-and-staging profile on.
> That'll be the maven dist process at play, nothing else
>
> On 4 Apr 2018, at 04:13, Wangda Tan <wheeleast@gmail.com<mailto:wh
> eeleast@gmail.com>> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org<ma...@apache.org>>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com<mailto:wh
> eeleast@gmail.com>> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com<
> mailto:aagarwal@hortonworks.com>>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagarwal@hortonworks.com<mailto:aagarwal@hortonworks.
> com>>
> Cc: Gera Shegalov <ge...@gmail.com>>, Sunil G <
> sunilg@apache.org<ma...@apache.org>>, "
> yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
> yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop
> Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org
> >>,
> "mapreduce-dev@hadoop.apache.org<ma...@hadoop.apache.org>"
> <ma...@hadoop.apache.org>>,
> Vinod Kumar Vavilapalli <vi...@apache.org>>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org<h
> ttp://repository.apache.org/> at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<
> http://3cyarn-dev.hadoop.apache.org/>
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<
> http://3cyarn-dev.hadoop.apache.org/>
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Suma Shivaprasad <su...@gmail.com>.
+1 (non binding)
*Verified - User Group Queue mapping - Node labels with New UI- Dynamic
queuesThanksSuma*
On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran <st...@hortonworks.com>
wrote:
> that's "dangerously interesting". I think you are right, and I also think
> it'll just be the version files which get generated
>
> anyway, +1 binding
>
> * ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer
> binding, then my downstream tests. All is well, provided you also have a
> spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version.
> That's an ongoing issue in the Spark project. With that JAR on my CP my
> downstream tests were all happy (yesterday)
>
> * today the staging files seem to be missing, at least maven is unable to
> find them even when I turn the spark snapshots-and-staging profile on.
> That'll be the maven dist process at play, nothing else
>
> On 4 Apr 2018, at 04:13, Wangda Tan <wheeleast@gmail.com<mailto:wh
> eeleast@gmail.com>> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org<ma...@apache.org>>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com<mailto:wh
> eeleast@gmail.com>> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com<
> mailto:aagarwal@hortonworks.com>>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagarwal@hortonworks.com<mailto:aagarwal@hortonworks.
> com>>
> Cc: Gera Shegalov <ge...@gmail.com>>, Sunil G <
> sunilg@apache.org<ma...@apache.org>>, "
> yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
> yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop
> Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org
> >>,
> "mapreduce-dev@hadoop.apache.org<ma...@hadoop.apache.org>"
> <ma...@hadoop.apache.org>>,
> Vinod Kumar Vavilapalli <vi...@apache.org>>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org<h
> ttp://repository.apache.org/> at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<
> http://3cyarn-dev.hadoop.apache.org/>
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<
> http://3cyarn-dev.hadoop.apache.org/>
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Suma Shivaprasad <su...@gmail.com>.
+1 (non binding)
*Verified - User Group Queue mapping - Node labels with New UI- Dynamic
queuesThanksSuma*
On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran <st...@hortonworks.com>
wrote:
> that's "dangerously interesting". I think you are right, and I also think
> it'll just be the version files which get generated
>
> anyway, +1 binding
>
> * ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer
> binding, then my downstream tests. All is well, provided you also have a
> spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version.
> That's an ongoing issue in the Spark project. With that JAR on my CP my
> downstream tests were all happy (yesterday)
>
> * today the staging files seem to be missing, at least maven is unable to
> find them even when I turn the spark snapshots-and-staging profile on.
> That'll be the maven dist process at play, nothing else
>
> On 4 Apr 2018, at 04:13, Wangda Tan <wheeleast@gmail.com<mailto:wh
> eeleast@gmail.com>> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org<ma...@apache.org>>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com<mailto:wh
> eeleast@gmail.com>> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com<
> mailto:aagarwal@hortonworks.com>>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagarwal@hortonworks.com<mailto:aagarwal@hortonworks.
> com>>
> Cc: Gera Shegalov <ge...@gmail.com>>, Sunil G <
> sunilg@apache.org<ma...@apache.org>>, "
> yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
> yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop
> Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org
> >>,
> "mapreduce-dev@hadoop.apache.org<ma...@hadoop.apache.org>"
> <ma...@hadoop.apache.org>>,
> Vinod Kumar Vavilapalli <vi...@apache.org>>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org<h
> ttp://repository.apache.org/> at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<
> http://3cyarn-dev.hadoop.apache.org/>
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<
> http://3cyarn-dev.hadoop.apache.org/>
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Steve Loughran <st...@hortonworks.com>.
that's "dangerously interesting". I think you are right, and I also think it'll just be the version files which get generated
anyway, +1 binding
* ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer binding, then my downstream tests. All is well, provided you also have a spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version. That's an ongoing issue in the Spark project. With that JAR on my CP my downstream tests were all happy (yesterday)
* today the staging files seem to be missing, at least maven is unable to find them even when I turn the spark snapshots-and-staging profile on. That'll be the maven dist process at play, nothing else
On 4 Apr 2018, at 04:13, Wangda Tan <wh...@gmail.com>> wrote:
Hi Vinod / Arpit,
I checked following versions:
- 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
Jars in maven repo [1] are *always* different from jars in the binary
tarball [2]: (I only checked hadoop-yarn-api-version.jar)
(Following numbers are sizes of the jar)
2.6.5:
- Jar in Maven: 1896185
- Jar in tarball: 1891485
2.7.5:
- Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
- Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
(Even size is same, md5 is different)
2.8.3:
- Jar in Maven: 2451433
- Jar in tarball: 2438975
2.9.0:
- Jar in Maven: 2791477
- Jar in tarball: 2777789
3.0.1:
- Jar in Maven: 2852604
- Jar in tarball: 2851373
I guess the differences come from our release process.
Thanks,
Wangda
[1] Maven jars are downloaded from
https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
<version>/hadoop-yarn-api-<version>.jar
[2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>>
wrote:
We vote on the source code. The binaries are convenience artifacts.
This is what I would do - (a) Just replace both the maven jars as well as
the binaries to be consistent and correct. And then (b) Give a couple more
days for folks who tested on the binaries to reverify - I count one such
clear vote as of now.
Thanks
+Vinod
On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com>> wrote:
HI Arpit,
I think it won't match if we do rebuild. It should be fine as far as
they're signed, correct? I don't see any policy doesn't allow this.
Thanks,
Wangda
On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>>
wrote:
Thanks Wangda, I see the shaded jars now.
Are the repo jars required to be the same as the binary release? They
don’t match right now, probably they got rebuilt.
+1 (binding), modulo that remaining question.
* Verified signatures
* Verified checksums for source and binary artefacts
* Sanity checked jars on r.a.o.
* Built from source
* Deployed to 3 node secure cluster with NameNode HA
* Verified HDFS web UIs
* Tried out HDFS shell commands
* Ran sample MapReduce jobs
Thanks!
----------------------------------------------------------------------
From: Wangda Tan <wh...@gmail.com>>
Date: Monday, April 2, 2018 at 9:25 PM
To: Arpit Agarwal <aa...@hortonworks.com>>
Cc: Gera Shegalov <ge...@gmail.com>>, Sunil G <su...@apache.org>>, "
yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <ya...@hadoop.apache.org>>, Hdfs-dev <
hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop Common <co...@hadoop.apache.org>>,
"mapreduce-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <ma...@hadoop.apache.org>>,
Vinod Kumar Vavilapalli <vi...@apache.org>>
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated
Maven repo? https://repository.apache.org/content/repositories/
orgapachehadoop-1092
Since the jars inside binary tarballs are correct (
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
need roll another RC, just update Maven repo should be sufficient.
Best,
Wangda
On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
wrote:
Hi Arpit,
Thanks for pointing out this.
I just removed all .md5 files from artifacts. I found md5 checksums still
exist in .mds files and I didn't remove them from .mds file because it is
generated by create-release script and Apache guidance is "should not"
instead of "must not". Please let me know if you think they need to be
removed as well.
- Wangda
On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
Thanks for putting together this RC, Wangda.
The guidance from Apache is to omit MD5s, specifically:
SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
https://www.apache.org/dev/release-distribution#sigs-and-sums
On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
wrote:
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
wrote:
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via http://repository.apache.org<http://repository.apache.org/> at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on
how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<http://3cyarn-dev.hadoop.apache.org/>
<http://3cyarn-dev.hadoop.apache.org/>%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<http://3cyarn-dev.hadoop.apache.org/>
<http://3cyarn-dev.hadoop.apache.org/>%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Steve Loughran <st...@hortonworks.com>.
that's "dangerously interesting". I think you are right, and I also think it'll just be the version files which get generated
anyway, +1 binding
* ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer binding, then my downstream tests. All is well, provided you also have a spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version. That's an ongoing issue in the Spark project. With that JAR on my CP my downstream tests were all happy (yesterday)
* today the staging files seem to be missing, at least maven is unable to find them even when I turn the spark snapshots-and-staging profile on. That'll be the maven dist process at play, nothing else
On 4 Apr 2018, at 04:13, Wangda Tan <wh...@gmail.com>> wrote:
Hi Vinod / Arpit,
I checked following versions:
- 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
Jars in maven repo [1] are *always* different from jars in the binary
tarball [2]: (I only checked hadoop-yarn-api-version.jar)
(Following numbers are sizes of the jar)
2.6.5:
- Jar in Maven: 1896185
- Jar in tarball: 1891485
2.7.5:
- Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
- Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
(Even size is same, md5 is different)
2.8.3:
- Jar in Maven: 2451433
- Jar in tarball: 2438975
2.9.0:
- Jar in Maven: 2791477
- Jar in tarball: 2777789
3.0.1:
- Jar in Maven: 2852604
- Jar in tarball: 2851373
I guess the differences come from our release process.
Thanks,
Wangda
[1] Maven jars are downloaded from
https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
<version>/hadoop-yarn-api-<version>.jar
[2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>>
wrote:
We vote on the source code. The binaries are convenience artifacts.
This is what I would do - (a) Just replace both the maven jars as well as
the binaries to be consistent and correct. And then (b) Give a couple more
days for folks who tested on the binaries to reverify - I count one such
clear vote as of now.
Thanks
+Vinod
On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com>> wrote:
HI Arpit,
I think it won't match if we do rebuild. It should be fine as far as
they're signed, correct? I don't see any policy doesn't allow this.
Thanks,
Wangda
On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>>
wrote:
Thanks Wangda, I see the shaded jars now.
Are the repo jars required to be the same as the binary release? They
don’t match right now, probably they got rebuilt.
+1 (binding), modulo that remaining question.
* Verified signatures
* Verified checksums for source and binary artefacts
* Sanity checked jars on r.a.o.
* Built from source
* Deployed to 3 node secure cluster with NameNode HA
* Verified HDFS web UIs
* Tried out HDFS shell commands
* Ran sample MapReduce jobs
Thanks!
----------------------------------------------------------------------
From: Wangda Tan <wh...@gmail.com>>
Date: Monday, April 2, 2018 at 9:25 PM
To: Arpit Agarwal <aa...@hortonworks.com>>
Cc: Gera Shegalov <ge...@gmail.com>>, Sunil G <su...@apache.org>>, "
yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <ya...@hadoop.apache.org>>, Hdfs-dev <
hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop Common <co...@hadoop.apache.org>>,
"mapreduce-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <ma...@hadoop.apache.org>>,
Vinod Kumar Vavilapalli <vi...@apache.org>>
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated
Maven repo? https://repository.apache.org/content/repositories/
orgapachehadoop-1092
Since the jars inside binary tarballs are correct (
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
need roll another RC, just update Maven repo should be sufficient.
Best,
Wangda
On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
wrote:
Hi Arpit,
Thanks for pointing out this.
I just removed all .md5 files from artifacts. I found md5 checksums still
exist in .mds files and I didn't remove them from .mds file because it is
generated by create-release script and Apache guidance is "should not"
instead of "must not". Please let me know if you think they need to be
removed as well.
- Wangda
On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
Thanks for putting together this RC, Wangda.
The guidance from Apache is to omit MD5s, specifically:
SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
https://www.apache.org/dev/release-distribution#sigs-and-sums
On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
wrote:
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
wrote:
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via http://repository.apache.org<http://repository.apache.org/> at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on
how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<http://3cyarn-dev.hadoop.apache.org/>
<http://3cyarn-dev.hadoop.apache.org/>%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<http://3cyarn-dev.hadoop.apache.org/>
<http://3cyarn-dev.hadoop.apache.org/>%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Gour Saha <gs...@hortonworks.com>.
Sorry for the really late send.
+1 (non-binding)
-Verified checksums and sha1
-Built from source and ran YARN Service core and api tests successfully
-Installed in a 7-node cluster
-Launched, flexed up/down, stopped, started and destroyed the sample sleeper service
-Launched, flexed, stopped, started Hive/LLAP
-Tested the Service REST APIs and command line
-Gour
On 4/5/18, 10:49 AM, "Wangda Tan" <wh...@gmail.com> wrote:
Thanks everybody for voting!
The vote passes successfully with 11 binding +1 votes, 8 non-binding +1
votes and no -1s.
I will work on the staging and releases.
Best,
Wangda Tan
On Thu, Apr 5, 2018 at 10:46 AM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
> That is a great observation. And I missed your previous email about the
> shaded vs unshaded jars already getting fixed.
>
> I guess we are good to go.
>
> ------------------------------------------------------------
> --------------------------------------------------------------
>
> Looking at the RC. Went through my usual check-list. Here's my summary.
>
> Verification
> - [Check] Successful recompilation from source tar-ball
> - [Check] Signature verification
> -- Note: The format of the mds files changed a bit - not a biggie.
> -- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "
> *hadoop-3.0.0-src.tar.gz*: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279
> 7BCED50B B2600318 B7E0BE3A 74DFFF71"
> -- But in 3.1.0 RC it is, "
> */build/source/target/artifacts/hadoop-3.1.0.tar.gz*: SHA256 = 670D2CED
> 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
> - [Check] Generating dist tarballs from source tar-ball
> - [Check] Testing
> -- Start NN, DN, RM, NM, JHS, Timeline Service
> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
> grep, pi
> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>
> +1 binding.
>
> Thanks
> +Vinod
>
> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
>
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aa...@hortonworks.com>
> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
>
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
>
>
>
>
>
>
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Gour Saha <gs...@hortonworks.com>.
Sorry for the really late send.
+1 (non-binding)
-Verified checksums and sha1
-Built from source and ran YARN Service core and api tests successfully
-Installed in a 7-node cluster
-Launched, flexed up/down, stopped, started and destroyed the sample sleeper service
-Launched, flexed, stopped, started Hive/LLAP
-Tested the Service REST APIs and command line
-Gour
On 4/5/18, 10:49 AM, "Wangda Tan" <wh...@gmail.com> wrote:
Thanks everybody for voting!
The vote passes successfully with 11 binding +1 votes, 8 non-binding +1
votes and no -1s.
I will work on the staging and releases.
Best,
Wangda Tan
On Thu, Apr 5, 2018 at 10:46 AM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
> That is a great observation. And I missed your previous email about the
> shaded vs unshaded jars already getting fixed.
>
> I guess we are good to go.
>
> ------------------------------------------------------------
> --------------------------------------------------------------
>
> Looking at the RC. Went through my usual check-list. Here's my summary.
>
> Verification
> - [Check] Successful recompilation from source tar-ball
> - [Check] Signature verification
> -- Note: The format of the mds files changed a bit - not a biggie.
> -- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "
> *hadoop-3.0.0-src.tar.gz*: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279
> 7BCED50B B2600318 B7E0BE3A 74DFFF71"
> -- But in 3.1.0 RC it is, "
> */build/source/target/artifacts/hadoop-3.1.0.tar.gz*: SHA256 = 670D2CED
> 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
> - [Check] Generating dist tarballs from source tar-ball
> - [Check] Testing
> -- Start NN, DN, RM, NM, JHS, Timeline Service
> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
> grep, pi
> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>
> +1 binding.
>
> Thanks
> +Vinod
>
> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
>
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aa...@hortonworks.com>
> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
>
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
>
>
>
>
>
>
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Gour Saha <gs...@hortonworks.com>.
Sorry for the really late send.
+1 (non-binding)
-Verified checksums and sha1
-Built from source and ran YARN Service core and api tests successfully
-Installed in a 7-node cluster
-Launched, flexed up/down, stopped, started and destroyed the sample sleeper service
-Launched, flexed, stopped, started Hive/LLAP
-Tested the Service REST APIs and command line
-Gour
On 4/5/18, 10:49 AM, "Wangda Tan" <wh...@gmail.com> wrote:
Thanks everybody for voting!
The vote passes successfully with 11 binding +1 votes, 8 non-binding +1
votes and no -1s.
I will work on the staging and releases.
Best,
Wangda Tan
On Thu, Apr 5, 2018 at 10:46 AM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
> That is a great observation. And I missed your previous email about the
> shaded vs unshaded jars already getting fixed.
>
> I guess we are good to go.
>
> ------------------------------------------------------------
> --------------------------------------------------------------
>
> Looking at the RC. Went through my usual check-list. Here's my summary.
>
> Verification
> - [Check] Successful recompilation from source tar-ball
> - [Check] Signature verification
> -- Note: The format of the mds files changed a bit - not a biggie.
> -- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "
> *hadoop-3.0.0-src.tar.gz*: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279
> 7BCED50B B2600318 B7E0BE3A 74DFFF71"
> -- But in 3.1.0 RC it is, "
> */build/source/target/artifacts/hadoop-3.1.0.tar.gz*: SHA256 = 670D2CED
> 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
> - [Check] Generating dist tarballs from source tar-ball
> - [Check] Testing
> -- Start NN, DN, RM, NM, JHS, Timeline Service
> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
> grep, pi
> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>
> +1 binding.
>
> Thanks
> +Vinod
>
> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
>
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aa...@hortonworks.com>
> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
>
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
>
>
>
>
>
>
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Gour Saha <gs...@hortonworks.com>.
Sorry for the really late send.
+1 (non-binding)
-Verified checksums and sha1
-Built from source and ran YARN Service core and api tests successfully
-Installed in a 7-node cluster
-Launched, flexed up/down, stopped, started and destroyed the sample sleeper service
-Launched, flexed, stopped, started Hive/LLAP
-Tested the Service REST APIs and command line
-Gour
On 4/5/18, 10:49 AM, "Wangda Tan" <wh...@gmail.com> wrote:
Thanks everybody for voting!
The vote passes successfully with 11 binding +1 votes, 8 non-binding +1
votes and no -1s.
I will work on the staging and releases.
Best,
Wangda Tan
On Thu, Apr 5, 2018 at 10:46 AM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
> That is a great observation. And I missed your previous email about the
> shaded vs unshaded jars already getting fixed.
>
> I guess we are good to go.
>
> ------------------------------------------------------------
> --------------------------------------------------------------
>
> Looking at the RC. Went through my usual check-list. Here's my summary.
>
> Verification
> - [Check] Successful recompilation from source tar-ball
> - [Check] Signature verification
> -- Note: The format of the mds files changed a bit - not a biggie.
> -- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "
> *hadoop-3.0.0-src.tar.gz*: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279
> 7BCED50B B2600318 B7E0BE3A 74DFFF71"
> -- But in 3.1.0 RC it is, "
> */build/source/target/artifacts/hadoop-3.1.0.tar.gz*: SHA256 = 670D2CED
> 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
> - [Check] Generating dist tarballs from source tar-ball
> - [Check] Testing
> -- Start NN, DN, RM, NM, JHS, Timeline Service
> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
> grep, pi
> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>
> +1 binding.
>
> Thanks
> +Vinod
>
> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
>
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aa...@hortonworks.com>
> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
>
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
>
>
>
>
>
>
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Thanks everybody for voting!
The vote passes successfully with 11 binding +1 votes, 8 non-binding +1
votes and no -1s.
I will work on the staging and releases.
Best,
Wangda Tan
On Thu, Apr 5, 2018 at 10:46 AM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
> That is a great observation. And I missed your previous email about the
> shaded vs unshaded jars already getting fixed.
>
> I guess we are good to go.
>
> ------------------------------------------------------------
> --------------------------------------------------------------
>
> Looking at the RC. Went through my usual check-list. Here's my summary.
>
> Verification
> - [Check] Successful recompilation from source tar-ball
> - [Check] Signature verification
> -- Note: The format of the mds files changed a bit - not a biggie.
> -- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "
> *hadoop-3.0.0-src.tar.gz*: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279
> 7BCED50B B2600318 B7E0BE3A 74DFFF71"
> -- But in 3.1.0 RC it is, "
> */build/source/target/artifacts/hadoop-3.1.0.tar.gz*: SHA256 = 670D2CED
> 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
> - [Check] Generating dist tarballs from source tar-ball
> - [Check] Testing
> -- Start NN, DN, RM, NM, JHS, Timeline Service
> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
> grep, pi
> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>
> +1 binding.
>
> Thanks
> +Vinod
>
> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
>
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aa...@hortonworks.com>
> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
>
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Thanks everybody for voting!
The vote passes successfully with 11 binding +1 votes, 8 non-binding +1
votes and no -1s.
I will work on the staging and releases.
Best,
Wangda Tan
On Thu, Apr 5, 2018 at 10:46 AM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
> That is a great observation. And I missed your previous email about the
> shaded vs unshaded jars already getting fixed.
>
> I guess we are good to go.
>
> ------------------------------------------------------------
> --------------------------------------------------------------
>
> Looking at the RC. Went through my usual check-list. Here's my summary.
>
> Verification
> - [Check] Successful recompilation from source tar-ball
> - [Check] Signature verification
> -- Note: The format of the mds files changed a bit - not a biggie.
> -- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "
> *hadoop-3.0.0-src.tar.gz*: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279
> 7BCED50B B2600318 B7E0BE3A 74DFFF71"
> -- But in 3.1.0 RC it is, "
> */build/source/target/artifacts/hadoop-3.1.0.tar.gz*: SHA256 = 670D2CED
> 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
> - [Check] Generating dist tarballs from source tar-ball
> - [Check] Testing
> -- Start NN, DN, RM, NM, JHS, Timeline Service
> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
> grep, pi
> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>
> +1 binding.
>
> Thanks
> +Vinod
>
> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
>
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aa...@hortonworks.com>
> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
>
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Thanks everybody for voting!
The vote passes successfully with 11 binding +1 votes, 8 non-binding +1
votes and no -1s.
I will work on the staging and releases.
Best,
Wangda Tan
On Thu, Apr 5, 2018 at 10:46 AM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
> That is a great observation. And I missed your previous email about the
> shaded vs unshaded jars already getting fixed.
>
> I guess we are good to go.
>
> ------------------------------------------------------------
> --------------------------------------------------------------
>
> Looking at the RC. Went through my usual check-list. Here's my summary.
>
> Verification
> - [Check] Successful recompilation from source tar-ball
> - [Check] Signature verification
> -- Note: The format of the mds files changed a bit - not a biggie.
> -- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "
> *hadoop-3.0.0-src.tar.gz*: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279
> 7BCED50B B2600318 B7E0BE3A 74DFFF71"
> -- But in 3.1.0 RC it is, "
> */build/source/target/artifacts/hadoop-3.1.0.tar.gz*: SHA256 = 670D2CED
> 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
> - [Check] Generating dist tarballs from source tar-ball
> - [Check] Testing
> -- Start NN, DN, RM, NM, JHS, Timeline Service
> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
> grep, pi
> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>
> +1 binding.
>
> Thanks
> +Vinod
>
> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
>
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aa...@hortonworks.com>
> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
>
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Thanks everybody for voting!
The vote passes successfully with 11 binding +1 votes, 8 non-binding +1
votes and no -1s.
I will work on the staging and releases.
Best,
Wangda Tan
On Thu, Apr 5, 2018 at 10:46 AM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
> That is a great observation. And I missed your previous email about the
> shaded vs unshaded jars already getting fixed.
>
> I guess we are good to go.
>
> ------------------------------------------------------------
> --------------------------------------------------------------
>
> Looking at the RC. Went through my usual check-list. Here's my summary.
>
> Verification
> - [Check] Successful recompilation from source tar-ball
> - [Check] Signature verification
> -- Note: The format of the mds files changed a bit - not a biggie.
> -- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "
> *hadoop-3.0.0-src.tar.gz*: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279
> 7BCED50B B2600318 B7E0BE3A 74DFFF71"
> -- But in 3.1.0 RC it is, "
> */build/source/target/artifacts/hadoop-3.1.0.tar.gz*: SHA256 = 670D2CED
> 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
> - [Check] Generating dist tarballs from source tar-ball
> - [Check] Testing
> -- Start NN, DN, RM, NM, JHS, Timeline Service
> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
> grep, pi
> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>
> +1 binding.
>
> Thanks
> +Vinod
>
> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
>
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aa...@hortonworks.com>
> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
>
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
> <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
That is a great observation. And I missed your previous email about the shaded vs unshaded jars already getting fixed.
I guess we are good to go.
--------------------------------------------------------------------------------------------------------------------------
Looking at the RC. Went through my usual check-list. Here's my summary.
Verification
- [Check] Successful recompilation from source tar-ball
- [Check] Signature verification
-- Note: The format of the mds files changed a bit - not a biggie.
-- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "hadoop-3.0.0-src.tar.gz: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279 7BCED50B B2600318 B7E0BE3A 74DFFF71"
-- But in 3.1.0 RC it is, "/build/source/target/artifacts/hadoop-3.1.0.tar.gz: SHA256 = 670D2CED 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
- [Check] Generating dist tarballs from source tar-ball
- [Check] Testing
-- Start NN, DN, RM, NM, JHS, Timeline Service
-- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
-- Tested CLIs to print nodes, apps etc and also navigated UIs
+1 binding.
Thanks
+Vinod
> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
> wrote:
>
>> We vote on the source code. The binaries are convenience artifacts.
>>
>> This is what I would do - (a) Just replace both the maven jars as well as
>> the binaries to be consistent and correct. And then (b) Give a couple more
>> days for folks who tested on the binaries to reverify - I count one such
>> clear vote as of now.
>>
>> Thanks
>> +Vinod
>>
>>
>> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>>
>> HI Arpit,
>>
>> I think it won't match if we do rebuild. It should be fine as far as
>> they're signed, correct? I don't see any policy doesn't allow this.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
>> wrote:
>>
>>> Thanks Wangda, I see the shaded jars now.
>>>
>>> Are the repo jars required to be the same as the binary release? They
>>> don’t match right now, probably they got rebuilt.
>>>
>>> +1 (binding), modulo that remaining question.
>>>
>>> * Verified signatures
>>> * Verified checksums for source and binary artefacts
>>> * Sanity checked jars on r.a.o.
>>> * Built from source
>>> * Deployed to 3 node secure cluster with NameNode HA
>>> * Verified HDFS web UIs
>>> * Tried out HDFS shell commands
>>> * Ran sample MapReduce jobs
>>>
>>> Thanks!
>>>
>>>
>>> ----------------------------------------------------------------------
>>> From: Wangda Tan <wh...@gmail.com>
>>> Date: Monday, April 2, 2018 at 9:25 PM
>>> To: Arpit Agarwal <aa...@hortonworks.com>
>>> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
>>> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
>>> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
>>> Vinod Kumar Vavilapalli <vi...@apache.org>
>>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>>
>>> As pointed by Arpit, the previously deployed shared jars are incorrect.
>>> Just redeployed jars and staged. @Arpit, could you please check the updated
>>> Maven repo? https://repository.apache.org/content/repositories/
>>> orgapachehadoop-1092
>>>
>>> Since the jars inside binary tarballs are correct (
>>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
>>> need roll another RC, just update Maven repo should be sufficient.
>>>
>>> Best,
>>> Wangda
>>>
>>>
>>> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
>>> wrote:
>>> Hi Arpit,
>>>
>>> Thanks for pointing out this.
>>>
>>> I just removed all .md5 files from artifacts. I found md5 checksums still
>>> exist in .mds files and I didn't remove them from .mds file because it is
>>> generated by create-release script and Apache guidance is "should not"
>>> instead of "must not". Please let me know if you think they need to be
>>> removed as well.
>>>
>>> - Wangda
>>>
>>>
>>>
>>> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
>>> aagarwal@hortonworks.com> wrote:
>>> Thanks for putting together this RC, Wangda.
>>>
>>> The guidance from Apache is to omit MD5s, specifically:
>>>> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>>
>>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>>
>>>
>>>
>>>
>>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
>>> wrote:
>>>
>>> Hi Gera,
>>>
>>> It's my bad, I thought only src/bin tarball is enough.
>>>
>>> I just uploaded all other things under artifact/ to
>>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>>
>>> Please let me know if you have any other comments.
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
>>> wrote:
>>>
>>>
>>> Thanks, Wangda!
>>>
>>> There are many more artifacts in previous votes, e.g., see
>>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>>> site tarball is missing.
>>>
>>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>>>
>>>
>>> Thanks Wangda for initiating the release.
>>>
>>> I tested this RC built from source file.
>>>
>>>
>>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>>> UI.
>>> - Below feature sanity is done
>>> - Application priority
>>> - Application timeout
>>> - Intra Queue preemption with priority based
>>> - DS based affinity tests to verify placement constraints.
>>> - Tested basic NodeLabel scenarios.
>>> - Added couple of labels to few of nodes and behavior is coming
>>> correct.
>>> - Verified old UI and new YARN UI for labels.
>>> - Submitted apps to labelled cluster and it works fine.
>>> - Also performed few cli commands related to nodelabel.
>>> - Test basic HA cases and seems correct.
>>> - Tested new YARN UI . All pages are getting loaded correctly.
>>>
>>>
>>> - Sunil
>>>
>>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
>>> wrote:
>>>
>>>
>>> Hi folks,
>>>
>>> Thanks to the many who helped with this release since Dec 2017 [1].
>>> We've
>>>
>>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>>
>>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>>
>>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>>
>>> The maven artifacts are available via http://repository.apache.org at
>>> https://repository.apache.org/content/repositories/
>>> orgapachehadoop-1090/
>>>
>>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>>
>>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>>> include the first class GPU/FPGA support on YARN, Native services,
>>> Support
>>>
>>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>>> block replicas to be provided by an external storage system, etc.
>>>
>>> For 3.1.0 RC0 vote discussion, please see [3].
>>>
>>> We’d like to use this as a starting release for 3.1.x [1], depending on
>>> how
>>>
>>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>>> as
>>>
>>> the stable release.
>>>
>>> We have done testing with a pseudo cluster:
>>> - Ran distributed job.
>>> - GPU scheduling/isolation.
>>> - Placement constraints (intra-application anti-affinity) by using
>>> distributed shell.
>>>
>>> My +1 to start.
>>>
>>> Best,
>>> Wangda/Vinod
>>>
>>> [1]
>>>
>>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
>>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>>
>>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>>> BY
>>>
>>> fixVersion ASC
>>> [3]
>>>
>>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
>>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Steve Loughran <st...@hortonworks.com>.
that's "dangerously interesting". I think you are right, and I also think it'll just be the version files which get generated
anyway, +1 binding
* ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer binding, then my downstream tests. All is well, provided you also have a spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version. That's an ongoing issue in the Spark project. With that JAR on my CP my downstream tests were all happy (yesterday)
* today the staging files seem to be missing, at least maven is unable to find them even when I turn the spark snapshots-and-staging profile on. That'll be the maven dist process at play, nothing else
On 4 Apr 2018, at 04:13, Wangda Tan <wh...@gmail.com>> wrote:
Hi Vinod / Arpit,
I checked following versions:
- 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
Jars in maven repo [1] are *always* different from jars in the binary
tarball [2]: (I only checked hadoop-yarn-api-version.jar)
(Following numbers are sizes of the jar)
2.6.5:
- Jar in Maven: 1896185
- Jar in tarball: 1891485
2.7.5:
- Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
- Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
(Even size is same, md5 is different)
2.8.3:
- Jar in Maven: 2451433
- Jar in tarball: 2438975
2.9.0:
- Jar in Maven: 2791477
- Jar in tarball: 2777789
3.0.1:
- Jar in Maven: 2852604
- Jar in tarball: 2851373
I guess the differences come from our release process.
Thanks,
Wangda
[1] Maven jars are downloaded from
https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
<version>/hadoop-yarn-api-<version>.jar
[2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>>
wrote:
We vote on the source code. The binaries are convenience artifacts.
This is what I would do - (a) Just replace both the maven jars as well as
the binaries to be consistent and correct. And then (b) Give a couple more
days for folks who tested on the binaries to reverify - I count one such
clear vote as of now.
Thanks
+Vinod
On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com>> wrote:
HI Arpit,
I think it won't match if we do rebuild. It should be fine as far as
they're signed, correct? I don't see any policy doesn't allow this.
Thanks,
Wangda
On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>>
wrote:
Thanks Wangda, I see the shaded jars now.
Are the repo jars required to be the same as the binary release? They
don’t match right now, probably they got rebuilt.
+1 (binding), modulo that remaining question.
* Verified signatures
* Verified checksums for source and binary artefacts
* Sanity checked jars on r.a.o.
* Built from source
* Deployed to 3 node secure cluster with NameNode HA
* Verified HDFS web UIs
* Tried out HDFS shell commands
* Ran sample MapReduce jobs
Thanks!
----------------------------------------------------------------------
From: Wangda Tan <wh...@gmail.com>>
Date: Monday, April 2, 2018 at 9:25 PM
To: Arpit Agarwal <aa...@hortonworks.com>>
Cc: Gera Shegalov <ge...@gmail.com>>, Sunil G <su...@apache.org>>, "
yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <ya...@hadoop.apache.org>>, Hdfs-dev <
hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop Common <co...@hadoop.apache.org>>,
"mapreduce-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <ma...@hadoop.apache.org>>,
Vinod Kumar Vavilapalli <vi...@apache.org>>
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated
Maven repo? https://repository.apache.org/content/repositories/
orgapachehadoop-1092
Since the jars inside binary tarballs are correct (
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
need roll another RC, just update Maven repo should be sufficient.
Best,
Wangda
On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
wrote:
Hi Arpit,
Thanks for pointing out this.
I just removed all .md5 files from artifacts. I found md5 checksums still
exist in .mds files and I didn't remove them from .mds file because it is
generated by create-release script and Apache guidance is "should not"
instead of "must not". Please let me know if you think they need to be
removed as well.
- Wangda
On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
Thanks for putting together this RC, Wangda.
The guidance from Apache is to omit MD5s, specifically:
SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
https://www.apache.org/dev/release-distribution#sigs-and-sums
On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
wrote:
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
wrote:
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via http://repository.apache.org<http://repository.apache.org/> at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on
how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<http://3cyarn-dev.hadoop.apache.org/>
<http://3cyarn-dev.hadoop.apache.org/>%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<http://3cyarn-dev.hadoop.apache.org/>
<http://3cyarn-dev.hadoop.apache.org/>%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Steve Loughran <st...@hortonworks.com>.
that's "dangerously interesting". I think you are right, and I also think it'll just be the version files which get generated
anyway, +1 binding
* ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer binding, then my downstream tests. All is well, provided you also have a spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version. That's an ongoing issue in the Spark project. With that JAR on my CP my downstream tests were all happy (yesterday)
* today the staging files seem to be missing, at least maven is unable to find them even when I turn the spark snapshots-and-staging profile on. That'll be the maven dist process at play, nothing else
On 4 Apr 2018, at 04:13, Wangda Tan <wh...@gmail.com>> wrote:
Hi Vinod / Arpit,
I checked following versions:
- 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
Jars in maven repo [1] are *always* different from jars in the binary
tarball [2]: (I only checked hadoop-yarn-api-version.jar)
(Following numbers are sizes of the jar)
2.6.5:
- Jar in Maven: 1896185
- Jar in tarball: 1891485
2.7.5:
- Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
- Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
(Even size is same, md5 is different)
2.8.3:
- Jar in Maven: 2451433
- Jar in tarball: 2438975
2.9.0:
- Jar in Maven: 2791477
- Jar in tarball: 2777789
3.0.1:
- Jar in Maven: 2852604
- Jar in tarball: 2851373
I guess the differences come from our release process.
Thanks,
Wangda
[1] Maven jars are downloaded from
https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
<version>/hadoop-yarn-api-<version>.jar
[2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>>
wrote:
We vote on the source code. The binaries are convenience artifacts.
This is what I would do - (a) Just replace both the maven jars as well as
the binaries to be consistent and correct. And then (b) Give a couple more
days for folks who tested on the binaries to reverify - I count one such
clear vote as of now.
Thanks
+Vinod
On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com>> wrote:
HI Arpit,
I think it won't match if we do rebuild. It should be fine as far as
they're signed, correct? I don't see any policy doesn't allow this.
Thanks,
Wangda
On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>>
wrote:
Thanks Wangda, I see the shaded jars now.
Are the repo jars required to be the same as the binary release? They
don’t match right now, probably they got rebuilt.
+1 (binding), modulo that remaining question.
* Verified signatures
* Verified checksums for source and binary artefacts
* Sanity checked jars on r.a.o.
* Built from source
* Deployed to 3 node secure cluster with NameNode HA
* Verified HDFS web UIs
* Tried out HDFS shell commands
* Ran sample MapReduce jobs
Thanks!
----------------------------------------------------------------------
From: Wangda Tan <wh...@gmail.com>>
Date: Monday, April 2, 2018 at 9:25 PM
To: Arpit Agarwal <aa...@hortonworks.com>>
Cc: Gera Shegalov <ge...@gmail.com>>, Sunil G <su...@apache.org>>, "
yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <ya...@hadoop.apache.org>>, Hdfs-dev <
hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>, Hadoop Common <co...@hadoop.apache.org>>,
"mapreduce-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <ma...@hadoop.apache.org>>,
Vinod Kumar Vavilapalli <vi...@apache.org>>
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated
Maven repo? https://repository.apache.org/content/repositories/
orgapachehadoop-1092
Since the jars inside binary tarballs are correct (
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
need roll another RC, just update Maven repo should be sufficient.
Best,
Wangda
On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
wrote:
Hi Arpit,
Thanks for pointing out this.
I just removed all .md5 files from artifacts. I found md5 checksums still
exist in .mds files and I didn't remove them from .mds file because it is
generated by create-release script and Apache guidance is "should not"
instead of "must not". Please let me know if you think they need to be
removed as well.
- Wangda
On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
aagarwal@hortonworks.com<ma...@hortonworks.com>> wrote:
Thanks for putting together this RC, Wangda.
The guidance from Apache is to omit MD5s, specifically:
SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
https://www.apache.org/dev/release-distribution#sigs-and-sums
On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
wrote:
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
wrote:
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via http://repository.apache.org<http://repository.apache.org/> at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on
how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<http://3cyarn-dev.hadoop.apache.org/>
<http://3cyarn-dev.hadoop.apache.org/>%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<http://3cyarn-dev.hadoop.apache.org/>
<http://3cyarn-dev.hadoop.apache.org/>%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
That is a great observation. And I missed your previous email about the shaded vs unshaded jars already getting fixed.
I guess we are good to go.
--------------------------------------------------------------------------------------------------------------------------
Looking at the RC. Went through my usual check-list. Here's my summary.
Verification
- [Check] Successful recompilation from source tar-ball
- [Check] Signature verification
-- Note: The format of the mds files changed a bit - not a biggie.
-- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "hadoop-3.0.0-src.tar.gz: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279 7BCED50B B2600318 B7E0BE3A 74DFFF71"
-- But in 3.1.0 RC it is, "/build/source/target/artifacts/hadoop-3.1.0.tar.gz: SHA256 = 670D2CED 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
- [Check] Generating dist tarballs from source tar-ball
- [Check] Testing
-- Start NN, DN, RM, NM, JHS, Timeline Service
-- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
-- Tested CLIs to print nodes, apps etc and also navigated UIs
+1 binding.
Thanks
+Vinod
> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
> wrote:
>
>> We vote on the source code. The binaries are convenience artifacts.
>>
>> This is what I would do - (a) Just replace both the maven jars as well as
>> the binaries to be consistent and correct. And then (b) Give a couple more
>> days for folks who tested on the binaries to reverify - I count one such
>> clear vote as of now.
>>
>> Thanks
>> +Vinod
>>
>>
>> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>>
>> HI Arpit,
>>
>> I think it won't match if we do rebuild. It should be fine as far as
>> they're signed, correct? I don't see any policy doesn't allow this.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
>> wrote:
>>
>>> Thanks Wangda, I see the shaded jars now.
>>>
>>> Are the repo jars required to be the same as the binary release? They
>>> don’t match right now, probably they got rebuilt.
>>>
>>> +1 (binding), modulo that remaining question.
>>>
>>> * Verified signatures
>>> * Verified checksums for source and binary artefacts
>>> * Sanity checked jars on r.a.o.
>>> * Built from source
>>> * Deployed to 3 node secure cluster with NameNode HA
>>> * Verified HDFS web UIs
>>> * Tried out HDFS shell commands
>>> * Ran sample MapReduce jobs
>>>
>>> Thanks!
>>>
>>>
>>> ----------------------------------------------------------------------
>>> From: Wangda Tan <wh...@gmail.com>
>>> Date: Monday, April 2, 2018 at 9:25 PM
>>> To: Arpit Agarwal <aa...@hortonworks.com>
>>> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
>>> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
>>> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
>>> Vinod Kumar Vavilapalli <vi...@apache.org>
>>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>>
>>> As pointed by Arpit, the previously deployed shared jars are incorrect.
>>> Just redeployed jars and staged. @Arpit, could you please check the updated
>>> Maven repo? https://repository.apache.org/content/repositories/
>>> orgapachehadoop-1092
>>>
>>> Since the jars inside binary tarballs are correct (
>>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
>>> need roll another RC, just update Maven repo should be sufficient.
>>>
>>> Best,
>>> Wangda
>>>
>>>
>>> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
>>> wrote:
>>> Hi Arpit,
>>>
>>> Thanks for pointing out this.
>>>
>>> I just removed all .md5 files from artifacts. I found md5 checksums still
>>> exist in .mds files and I didn't remove them from .mds file because it is
>>> generated by create-release script and Apache guidance is "should not"
>>> instead of "must not". Please let me know if you think they need to be
>>> removed as well.
>>>
>>> - Wangda
>>>
>>>
>>>
>>> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
>>> aagarwal@hortonworks.com> wrote:
>>> Thanks for putting together this RC, Wangda.
>>>
>>> The guidance from Apache is to omit MD5s, specifically:
>>>> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>>
>>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>>
>>>
>>>
>>>
>>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
>>> wrote:
>>>
>>> Hi Gera,
>>>
>>> It's my bad, I thought only src/bin tarball is enough.
>>>
>>> I just uploaded all other things under artifact/ to
>>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>>
>>> Please let me know if you have any other comments.
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
>>> wrote:
>>>
>>>
>>> Thanks, Wangda!
>>>
>>> There are many more artifacts in previous votes, e.g., see
>>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>>> site tarball is missing.
>>>
>>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>>>
>>>
>>> Thanks Wangda for initiating the release.
>>>
>>> I tested this RC built from source file.
>>>
>>>
>>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>>> UI.
>>> - Below feature sanity is done
>>> - Application priority
>>> - Application timeout
>>> - Intra Queue preemption with priority based
>>> - DS based affinity tests to verify placement constraints.
>>> - Tested basic NodeLabel scenarios.
>>> - Added couple of labels to few of nodes and behavior is coming
>>> correct.
>>> - Verified old UI and new YARN UI for labels.
>>> - Submitted apps to labelled cluster and it works fine.
>>> - Also performed few cli commands related to nodelabel.
>>> - Test basic HA cases and seems correct.
>>> - Tested new YARN UI . All pages are getting loaded correctly.
>>>
>>>
>>> - Sunil
>>>
>>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
>>> wrote:
>>>
>>>
>>> Hi folks,
>>>
>>> Thanks to the many who helped with this release since Dec 2017 [1].
>>> We've
>>>
>>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>>
>>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>>
>>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>>
>>> The maven artifacts are available via http://repository.apache.org at
>>> https://repository.apache.org/content/repositories/
>>> orgapachehadoop-1090/
>>>
>>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>>
>>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>>> include the first class GPU/FPGA support on YARN, Native services,
>>> Support
>>>
>>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>>> block replicas to be provided by an external storage system, etc.
>>>
>>> For 3.1.0 RC0 vote discussion, please see [3].
>>>
>>> We’d like to use this as a starting release for 3.1.x [1], depending on
>>> how
>>>
>>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>>> as
>>>
>>> the stable release.
>>>
>>> We have done testing with a pseudo cluster:
>>> - Ran distributed job.
>>> - GPU scheduling/isolation.
>>> - Placement constraints (intra-application anti-affinity) by using
>>> distributed shell.
>>>
>>> My +1 to start.
>>>
>>> Best,
>>> Wangda/Vinod
>>>
>>> [1]
>>>
>>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
>>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>>
>>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>>> BY
>>>
>>> fixVersion ASC
>>> [3]
>>>
>>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
>>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
That is a great observation. And I missed your previous email about the shaded vs unshaded jars already getting fixed.
I guess we are good to go.
--------------------------------------------------------------------------------------------------------------------------
Looking at the RC. Went through my usual check-list. Here's my summary.
Verification
- [Check] Successful recompilation from source tar-ball
- [Check] Signature verification
-- Note: The format of the mds files changed a bit - not a biggie.
-- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "hadoop-3.0.0-src.tar.gz: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279 7BCED50B B2600318 B7E0BE3A 74DFFF71"
-- But in 3.1.0 RC it is, "/build/source/target/artifacts/hadoop-3.1.0.tar.gz: SHA256 = 670D2CED 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
- [Check] Generating dist tarballs from source tar-ball
- [Check] Testing
-- Start NN, DN, RM, NM, JHS, Timeline Service
-- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
-- Tested CLIs to print nodes, apps etc and also navigated UIs
+1 binding.
Thanks
+Vinod
> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
> wrote:
>
>> We vote on the source code. The binaries are convenience artifacts.
>>
>> This is what I would do - (a) Just replace both the maven jars as well as
>> the binaries to be consistent and correct. And then (b) Give a couple more
>> days for folks who tested on the binaries to reverify - I count one such
>> clear vote as of now.
>>
>> Thanks
>> +Vinod
>>
>>
>> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>>
>> HI Arpit,
>>
>> I think it won't match if we do rebuild. It should be fine as far as
>> they're signed, correct? I don't see any policy doesn't allow this.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
>> wrote:
>>
>>> Thanks Wangda, I see the shaded jars now.
>>>
>>> Are the repo jars required to be the same as the binary release? They
>>> don’t match right now, probably they got rebuilt.
>>>
>>> +1 (binding), modulo that remaining question.
>>>
>>> * Verified signatures
>>> * Verified checksums for source and binary artefacts
>>> * Sanity checked jars on r.a.o.
>>> * Built from source
>>> * Deployed to 3 node secure cluster with NameNode HA
>>> * Verified HDFS web UIs
>>> * Tried out HDFS shell commands
>>> * Ran sample MapReduce jobs
>>>
>>> Thanks!
>>>
>>>
>>> ----------------------------------------------------------------------
>>> From: Wangda Tan <wh...@gmail.com>
>>> Date: Monday, April 2, 2018 at 9:25 PM
>>> To: Arpit Agarwal <aa...@hortonworks.com>
>>> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
>>> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
>>> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
>>> Vinod Kumar Vavilapalli <vi...@apache.org>
>>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>>
>>> As pointed by Arpit, the previously deployed shared jars are incorrect.
>>> Just redeployed jars and staged. @Arpit, could you please check the updated
>>> Maven repo? https://repository.apache.org/content/repositories/
>>> orgapachehadoop-1092
>>>
>>> Since the jars inside binary tarballs are correct (
>>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
>>> need roll another RC, just update Maven repo should be sufficient.
>>>
>>> Best,
>>> Wangda
>>>
>>>
>>> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
>>> wrote:
>>> Hi Arpit,
>>>
>>> Thanks for pointing out this.
>>>
>>> I just removed all .md5 files from artifacts. I found md5 checksums still
>>> exist in .mds files and I didn't remove them from .mds file because it is
>>> generated by create-release script and Apache guidance is "should not"
>>> instead of "must not". Please let me know if you think they need to be
>>> removed as well.
>>>
>>> - Wangda
>>>
>>>
>>>
>>> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
>>> aagarwal@hortonworks.com> wrote:
>>> Thanks for putting together this RC, Wangda.
>>>
>>> The guidance from Apache is to omit MD5s, specifically:
>>>> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>>
>>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>>
>>>
>>>
>>>
>>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
>>> wrote:
>>>
>>> Hi Gera,
>>>
>>> It's my bad, I thought only src/bin tarball is enough.
>>>
>>> I just uploaded all other things under artifact/ to
>>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>>
>>> Please let me know if you have any other comments.
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
>>> wrote:
>>>
>>>
>>> Thanks, Wangda!
>>>
>>> There are many more artifacts in previous votes, e.g., see
>>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>>> site tarball is missing.
>>>
>>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>>>
>>>
>>> Thanks Wangda for initiating the release.
>>>
>>> I tested this RC built from source file.
>>>
>>>
>>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>>> UI.
>>> - Below feature sanity is done
>>> - Application priority
>>> - Application timeout
>>> - Intra Queue preemption with priority based
>>> - DS based affinity tests to verify placement constraints.
>>> - Tested basic NodeLabel scenarios.
>>> - Added couple of labels to few of nodes and behavior is coming
>>> correct.
>>> - Verified old UI and new YARN UI for labels.
>>> - Submitted apps to labelled cluster and it works fine.
>>> - Also performed few cli commands related to nodelabel.
>>> - Test basic HA cases and seems correct.
>>> - Tested new YARN UI . All pages are getting loaded correctly.
>>>
>>>
>>> - Sunil
>>>
>>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
>>> wrote:
>>>
>>>
>>> Hi folks,
>>>
>>> Thanks to the many who helped with this release since Dec 2017 [1].
>>> We've
>>>
>>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>>
>>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>>
>>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>>
>>> The maven artifacts are available via http://repository.apache.org at
>>> https://repository.apache.org/content/repositories/
>>> orgapachehadoop-1090/
>>>
>>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>>
>>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>>> include the first class GPU/FPGA support on YARN, Native services,
>>> Support
>>>
>>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>>> block replicas to be provided by an external storage system, etc.
>>>
>>> For 3.1.0 RC0 vote discussion, please see [3].
>>>
>>> We’d like to use this as a starting release for 3.1.x [1], depending on
>>> how
>>>
>>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>>> as
>>>
>>> the stable release.
>>>
>>> We have done testing with a pseudo cluster:
>>> - Ran distributed job.
>>> - GPU scheduling/isolation.
>>> - Placement constraints (intra-application anti-affinity) by using
>>> distributed shell.
>>>
>>> My +1 to start.
>>>
>>> Best,
>>> Wangda/Vinod
>>>
>>> [1]
>>>
>>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
>>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>>
>>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>>> BY
>>>
>>> fixVersion ASC
>>> [3]
>>>
>>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
>>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
That is a great observation. And I missed your previous email about the shaded vs unshaded jars already getting fixed.
I guess we are good to go.
--------------------------------------------------------------------------------------------------------------------------
Looking at the RC. Went through my usual check-list. Here's my summary.
Verification
- [Check] Successful recompilation from source tar-ball
- [Check] Signature verification
-- Note: The format of the mds files changed a bit - not a biggie.
-- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "hadoop-3.0.0-src.tar.gz: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279 7BCED50B B2600318 B7E0BE3A 74DFFF71"
-- But in 3.1.0 RC it is, "/build/source/target/artifacts/hadoop-3.1.0.tar.gz: SHA256 = 670D2CED 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
- [Check] Generating dist tarballs from source tar-ball
- [Check] Testing
-- Start NN, DN, RM, NM, JHS, Timeline Service
-- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
-- Tested CLIs to print nodes, apps etc and also navigated UIs
+1 binding.
Thanks
+Vinod
> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 2777789
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
> <version>/hadoop-yarn-api-<version>.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
> wrote:
>
>> We vote on the source code. The binaries are convenience artifacts.
>>
>> This is what I would do - (a) Just replace both the maven jars as well as
>> the binaries to be consistent and correct. And then (b) Give a couple more
>> days for folks who tested on the binaries to reverify - I count one such
>> clear vote as of now.
>>
>> Thanks
>> +Vinod
>>
>>
>> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>>
>> HI Arpit,
>>
>> I think it won't match if we do rebuild. It should be fine as far as
>> they're signed, correct? I don't see any policy doesn't allow this.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
>> wrote:
>>
>>> Thanks Wangda, I see the shaded jars now.
>>>
>>> Are the repo jars required to be the same as the binary release? They
>>> don’t match right now, probably they got rebuilt.
>>>
>>> +1 (binding), modulo that remaining question.
>>>
>>> * Verified signatures
>>> * Verified checksums for source and binary artefacts
>>> * Sanity checked jars on r.a.o.
>>> * Built from source
>>> * Deployed to 3 node secure cluster with NameNode HA
>>> * Verified HDFS web UIs
>>> * Tried out HDFS shell commands
>>> * Ran sample MapReduce jobs
>>>
>>> Thanks!
>>>
>>>
>>> ----------------------------------------------------------------------
>>> From: Wangda Tan <wh...@gmail.com>
>>> Date: Monday, April 2, 2018 at 9:25 PM
>>> To: Arpit Agarwal <aa...@hortonworks.com>
>>> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
>>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
>>> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
>>> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
>>> Vinod Kumar Vavilapalli <vi...@apache.org>
>>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>>
>>> As pointed by Arpit, the previously deployed shared jars are incorrect.
>>> Just redeployed jars and staged. @Arpit, could you please check the updated
>>> Maven repo? https://repository.apache.org/content/repositories/
>>> orgapachehadoop-1092
>>>
>>> Since the jars inside binary tarballs are correct (
>>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
>>> need roll another RC, just update Maven repo should be sufficient.
>>>
>>> Best,
>>> Wangda
>>>
>>>
>>> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
>>> wrote:
>>> Hi Arpit,
>>>
>>> Thanks for pointing out this.
>>>
>>> I just removed all .md5 files from artifacts. I found md5 checksums still
>>> exist in .mds files and I didn't remove them from .mds file because it is
>>> generated by create-release script and Apache guidance is "should not"
>>> instead of "must not". Please let me know if you think they need to be
>>> removed as well.
>>>
>>> - Wangda
>>>
>>>
>>>
>>> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
>>> aagarwal@hortonworks.com> wrote:
>>> Thanks for putting together this RC, Wangda.
>>>
>>> The guidance from Apache is to omit MD5s, specifically:
>>>> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>>
>>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>>
>>>
>>>
>>>
>>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
>>> wrote:
>>>
>>> Hi Gera,
>>>
>>> It's my bad, I thought only src/bin tarball is enough.
>>>
>>> I just uploaded all other things under artifact/ to
>>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>>
>>> Please let me know if you have any other comments.
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
>>> wrote:
>>>
>>>
>>> Thanks, Wangda!
>>>
>>> There are many more artifacts in previous votes, e.g., see
>>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>>> site tarball is missing.
>>>
>>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>>>
>>>
>>> Thanks Wangda for initiating the release.
>>>
>>> I tested this RC built from source file.
>>>
>>>
>>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>>> UI.
>>> - Below feature sanity is done
>>> - Application priority
>>> - Application timeout
>>> - Intra Queue preemption with priority based
>>> - DS based affinity tests to verify placement constraints.
>>> - Tested basic NodeLabel scenarios.
>>> - Added couple of labels to few of nodes and behavior is coming
>>> correct.
>>> - Verified old UI and new YARN UI for labels.
>>> - Submitted apps to labelled cluster and it works fine.
>>> - Also performed few cli commands related to nodelabel.
>>> - Test basic HA cases and seems correct.
>>> - Tested new YARN UI . All pages are getting loaded correctly.
>>>
>>>
>>> - Sunil
>>>
>>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
>>> wrote:
>>>
>>>
>>> Hi folks,
>>>
>>> Thanks to the many who helped with this release since Dec 2017 [1].
>>> We've
>>>
>>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>>
>>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>>
>>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>>
>>> The maven artifacts are available via http://repository.apache.org at
>>> https://repository.apache.org/content/repositories/
>>> orgapachehadoop-1090/
>>>
>>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>>
>>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>>> include the first class GPU/FPGA support on YARN, Native services,
>>> Support
>>>
>>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>>> block replicas to be provided by an external storage system, etc.
>>>
>>> For 3.1.0 RC0 vote discussion, please see [3].
>>>
>>> We’d like to use this as a starting release for 3.1.x [1], depending on
>>> how
>>>
>>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>>> as
>>>
>>> the stable release.
>>>
>>> We have done testing with a pseudo cluster:
>>> - Ran distributed job.
>>> - GPU scheduling/isolation.
>>> - Placement constraints (intra-application anti-affinity) by using
>>> distributed shell.
>>>
>>> My +1 to start.
>>>
>>> Best,
>>> Wangda/Vinod
>>>
>>> [1]
>>>
>>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
>>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>>
>>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>>> BY
>>>
>>> fixVersion ASC
>>> [3]
>>>
>>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
>>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Hi Vinod / Arpit,
I checked following versions:
- 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
Jars in maven repo [1] are *always* different from jars in the binary
tarball [2]: (I only checked hadoop-yarn-api-version.jar)
(Following numbers are sizes of the jar)
2.6.5:
- Jar in Maven: 1896185
- Jar in tarball: 1891485
2.7.5:
- Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
- Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
(Even size is same, md5 is different)
2.8.3:
- Jar in Maven: 2451433
- Jar in tarball: 2438975
2.9.0:
- Jar in Maven: 2791477
- Jar in tarball: 2777789
3.0.1:
- Jar in Maven: 2852604
- Jar in tarball: 2851373
I guess the differences come from our release process.
Thanks,
Wangda
[1] Maven jars are downloaded from
https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
<version>/hadoop-yarn-api-<version>.jar
[2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
>> Thanks Wangda, I see the shaded jars now.
>>
>> Are the repo jars required to be the same as the binary release? They
>> don’t match right now, probably they got rebuilt.
>>
>> +1 (binding), modulo that remaining question.
>>
>> * Verified signatures
>> * Verified checksums for source and binary artefacts
>> * Sanity checked jars on r.a.o.
>> * Built from source
>> * Deployed to 3 node secure cluster with NameNode HA
>> * Verified HDFS web UIs
>> * Tried out HDFS shell commands
>> * Ran sample MapReduce jobs
>>
>> Thanks!
>>
>>
>> ----------------------------------------------------------------------
>> From: Wangda Tan <wh...@gmail.com>
>> Date: Monday, April 2, 2018 at 9:25 PM
>> To: Arpit Agarwal <aa...@hortonworks.com>
>> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
>> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
>> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
>> Vinod Kumar Vavilapalli <vi...@apache.org>
>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>
>> As pointed by Arpit, the previously deployed shared jars are incorrect.
>> Just redeployed jars and staged. @Arpit, could you please check the updated
>> Maven repo? https://repository.apache.org/content/repositories/
>> orgapachehadoop-1092
>>
>> Since the jars inside binary tarballs are correct (
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
>> need roll another RC, just update Maven repo should be sufficient.
>>
>> Best,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
>> wrote:
>> Hi Arpit,
>>
>> Thanks for pointing out this.
>>
>> I just removed all .md5 files from artifacts. I found md5 checksums still
>> exist in .mds files and I didn't remove them from .mds file because it is
>> generated by create-release script and Apache guidance is "should not"
>> instead of "must not". Please let me know if you think they need to be
>> removed as well.
>>
>> - Wangda
>>
>>
>>
>> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
>> aagarwal@hortonworks.com> wrote:
>> Thanks for putting together this RC, Wangda.
>>
>> The guidance from Apache is to omit MD5s, specifically:
>> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>>
>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
>> wrote:
>>
>> Hi Gera,
>>
>> It's my bad, I thought only src/bin tarball is enough.
>>
>> I just uploaded all other things under artifact/ to
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>> Please let me know if you have any other comments.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
>> wrote:
>>
>>
>> Thanks, Wangda!
>>
>> There are many more artifacts in previous votes, e.g., see
>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>> site tarball is missing.
>>
>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>>
>>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>> UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
>> wrote:
>>
>>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>>
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via http://repository.apache.org at
>> https://repository.apache.org/content/repositories/
>> orgapachehadoop-1090/
>>
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services,
>> Support
>>
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>>
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>> as
>>
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>>
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Hi Vinod / Arpit,
I checked following versions:
- 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
Jars in maven repo [1] are *always* different from jars in the binary
tarball [2]: (I only checked hadoop-yarn-api-version.jar)
(Following numbers are sizes of the jar)
2.6.5:
- Jar in Maven: 1896185
- Jar in tarball: 1891485
2.7.5:
- Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
- Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
(Even size is same, md5 is different)
2.8.3:
- Jar in Maven: 2451433
- Jar in tarball: 2438975
2.9.0:
- Jar in Maven: 2791477
- Jar in tarball: 2777789
3.0.1:
- Jar in Maven: 2852604
- Jar in tarball: 2851373
I guess the differences come from our release process.
Thanks,
Wangda
[1] Maven jars are downloaded from
https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
<version>/hadoop-yarn-api-<version>.jar
[2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
>> Thanks Wangda, I see the shaded jars now.
>>
>> Are the repo jars required to be the same as the binary release? They
>> don’t match right now, probably they got rebuilt.
>>
>> +1 (binding), modulo that remaining question.
>>
>> * Verified signatures
>> * Verified checksums for source and binary artefacts
>> * Sanity checked jars on r.a.o.
>> * Built from source
>> * Deployed to 3 node secure cluster with NameNode HA
>> * Verified HDFS web UIs
>> * Tried out HDFS shell commands
>> * Ran sample MapReduce jobs
>>
>> Thanks!
>>
>>
>> ----------------------------------------------------------------------
>> From: Wangda Tan <wh...@gmail.com>
>> Date: Monday, April 2, 2018 at 9:25 PM
>> To: Arpit Agarwal <aa...@hortonworks.com>
>> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
>> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
>> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
>> Vinod Kumar Vavilapalli <vi...@apache.org>
>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>
>> As pointed by Arpit, the previously deployed shared jars are incorrect.
>> Just redeployed jars and staged. @Arpit, could you please check the updated
>> Maven repo? https://repository.apache.org/content/repositories/
>> orgapachehadoop-1092
>>
>> Since the jars inside binary tarballs are correct (
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
>> need roll another RC, just update Maven repo should be sufficient.
>>
>> Best,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
>> wrote:
>> Hi Arpit,
>>
>> Thanks for pointing out this.
>>
>> I just removed all .md5 files from artifacts. I found md5 checksums still
>> exist in .mds files and I didn't remove them from .mds file because it is
>> generated by create-release script and Apache guidance is "should not"
>> instead of "must not". Please let me know if you think they need to be
>> removed as well.
>>
>> - Wangda
>>
>>
>>
>> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
>> aagarwal@hortonworks.com> wrote:
>> Thanks for putting together this RC, Wangda.
>>
>> The guidance from Apache is to omit MD5s, specifically:
>> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>>
>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
>> wrote:
>>
>> Hi Gera,
>>
>> It's my bad, I thought only src/bin tarball is enough.
>>
>> I just uploaded all other things under artifact/ to
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>> Please let me know if you have any other comments.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
>> wrote:
>>
>>
>> Thanks, Wangda!
>>
>> There are many more artifacts in previous votes, e.g., see
>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>> site tarball is missing.
>>
>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>>
>>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>> UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
>> wrote:
>>
>>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>>
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via http://repository.apache.org at
>> https://repository.apache.org/content/repositories/
>> orgapachehadoop-1090/
>>
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services,
>> Support
>>
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>>
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>> as
>>
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>>
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Iñigo Goiri <el...@gmail.com>.
+1 (non binding)
* Deployed with 4 subclusters with HDFS Router-based federation.
* Executed DistCp across subclusters through the Router
* Checked documentation and tgz
On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
> > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > HI Arpit,
> >
> > I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com
> <ma...@hortonworks.com>> wrote:
> > Thanks Wangda, I see the shaded jars now.
> >
> > Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
> >
> > +1 (binding), modulo that remaining question.
> >
> > * Verified signatures
> > * Verified checksums for source and binary artefacts
> > * Sanity checked jars on r.a.o.
> > * Built from source
> > * Deployed to 3 node secure cluster with NameNode HA
> > * Verified HDFS web UIs
> > * Tried out HDFS shell commands
> > * Ran sample MapReduce jobs
> >
> > Thanks!
> >
> >
> > ----------------------------------------------------------------------
> > From: Wangda Tan <wheeleast@gmail.com <ma...@gmail.com>>
> > Date: Monday, April 2, 2018 at 9:25 PM
> > To: Arpit Agarwal <aagarwal@hortonworks.com <mailto:
> aagarwal@hortonworks.com>>
> > Cc: Gera Shegalov <gera05@gmail.com <ma...@gmail.com>>, Sunil G
> <sunilg@apache.org <ma...@apache.org>>, "
> yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>" <
> yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hdfs-dev
> <hdfs-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hadoop
> Common <common-dev@hadoop.apache.org <ma...@hadoop.apache.org>>,
> "mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>"
> <mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>>,
> Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>>
> > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> >
> > As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092 <https://repository.apache.org/content/repositories/
> orgapachehadoop-1092>
> >
> > Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
> >
> > Best,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com
> <ma...@gmail.com>> wrote:
> > Hi Arpit,
> >
> > Thanks for pointing out this.
> >
> > I just removed all .md5 files from artifacts. I found md5 checksums
> still exist in .mds files and I didn't remove them from .mds file because
> it is generated by create-release script and Apache guidance is "should
> not" instead of "must not". Please let me know if you think they need to be
> removed as well.
> >
> > - Wangda
> >
> >
> >
> > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com <ma...@hortonworks.com>> wrote:
> > Thanks for putting together this RC, Wangda.
> >
> > The guidance from Apache is to omit MD5s, specifically:
> > > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> >
> > https://www.apache.org/dev/release-distribution#sigs-and-sums <
> https://www.apache.org/dev/release-distribution#sigs-and-sums>
> >
> >
> >
> >
> > On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheeleast@gmail.com
> <ma...@gmail.com>> wrote:
> >
> > Hi Gera,
> >
> > It's my bad, I thought only src/bin tarball is enough.
> >
> > I just uploaded all other things under artifact/ to
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>
> >
> > Please let me know if you have any other comments.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:gera05@gmail.com
> <ma...@gmail.com>> wrote:
> >
> >
> > Thanks, Wangda!
> >
> > There are many more artifacts in previous votes, e.g., see
> > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ <
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/> . Among others the
> > site tarball is missing.
> >
> > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sunilg@apache.org
> <ma...@apache.org>> wrote:
> >
> >
> > Thanks Wangda for initiating the release.
> >
> > I tested this RC built from source file.
> >
> >
> > - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> > UI.
> > - Below feature sanity is done
> > - Application priority
> > - Application timeout
> > - Intra Queue preemption with priority based
> > - DS based affinity tests to verify placement constraints.
> > - Tested basic NodeLabel scenarios.
> > - Added couple of labels to few of nodes and behavior is coming
> > correct.
> > - Verified old UI and new YARN UI for labels.
> > - Submitted apps to labelled cluster and it works fine.
> > - Also performed few cli commands related to nodelabel.
> > - Test basic HA cases and seems correct.
> > - Tested new YARN UI . All pages are getting loaded correctly.
> >
> >
> > - Sunil
> >
> > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com
> <ma...@gmail.com>> wrote:
> >
> >
> > Hi folks,
> >
> > Thanks to the many who helped with this release since Dec 2017 [1].
> > We've
> >
> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1 <
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1>
> >
> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >
> > The maven artifacts are available via http://repository.apache.org <
> http://repository.apache.org/> at
> > https://repository.apache.org/content/repositories/ <
> https://repository.apache.org/content/repositories/>
> > orgapachehadoop-1090/
> >
> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >
> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > include the first class GPU/FPGA support on YARN, Native services,
> > Support
> >
> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > block replicas to be provided by an external storage system, etc.
> >
> > For 3.1.0 RC0 vote discussion, please see [3].
> >
> > We’d like to use this as a starting release for 3.1.x [1], depending on
> > how
> >
> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> > as
> >
> > the stable release.
> >
> > We have done testing with a pseudo cluster:
> > - Ran distributed job.
> > - GPU scheduling/isolation.
> > - Placement constraints (intra-application anti-affinity) by using
> > distributed shell.
> >
> > My +1 to start.
> >
> > Best,
> > Wangda/Vinod
> >
> > [1]
> >
> > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b <
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b>
> > c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org <
> http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> > BY
> >
> > fixVersion ASC
> > [3]
> >
> > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7 <
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7>
> > 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org <
> http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Hi Vinod / Arpit,
I checked following versions:
- 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
Jars in maven repo [1] are *always* different from jars in the binary
tarball [2]: (I only checked hadoop-yarn-api-version.jar)
(Following numbers are sizes of the jar)
2.6.5:
- Jar in Maven: 1896185
- Jar in tarball: 1891485
2.7.5:
- Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
- Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
(Even size is same, md5 is different)
2.8.3:
- Jar in Maven: 2451433
- Jar in tarball: 2438975
2.9.0:
- Jar in Maven: 2791477
- Jar in tarball: 2777789
3.0.1:
- Jar in Maven: 2852604
- Jar in tarball: 2851373
I guess the differences come from our release process.
Thanks,
Wangda
[1] Maven jars are downloaded from
https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
<version>/hadoop-yarn-api-<version>.jar
[2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
>> Thanks Wangda, I see the shaded jars now.
>>
>> Are the repo jars required to be the same as the binary release? They
>> don’t match right now, probably they got rebuilt.
>>
>> +1 (binding), modulo that remaining question.
>>
>> * Verified signatures
>> * Verified checksums for source and binary artefacts
>> * Sanity checked jars on r.a.o.
>> * Built from source
>> * Deployed to 3 node secure cluster with NameNode HA
>> * Verified HDFS web UIs
>> * Tried out HDFS shell commands
>> * Ran sample MapReduce jobs
>>
>> Thanks!
>>
>>
>> ----------------------------------------------------------------------
>> From: Wangda Tan <wh...@gmail.com>
>> Date: Monday, April 2, 2018 at 9:25 PM
>> To: Arpit Agarwal <aa...@hortonworks.com>
>> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
>> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
>> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
>> Vinod Kumar Vavilapalli <vi...@apache.org>
>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>
>> As pointed by Arpit, the previously deployed shared jars are incorrect.
>> Just redeployed jars and staged. @Arpit, could you please check the updated
>> Maven repo? https://repository.apache.org/content/repositories/
>> orgapachehadoop-1092
>>
>> Since the jars inside binary tarballs are correct (
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
>> need roll another RC, just update Maven repo should be sufficient.
>>
>> Best,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
>> wrote:
>> Hi Arpit,
>>
>> Thanks for pointing out this.
>>
>> I just removed all .md5 files from artifacts. I found md5 checksums still
>> exist in .mds files and I didn't remove them from .mds file because it is
>> generated by create-release script and Apache guidance is "should not"
>> instead of "must not". Please let me know if you think they need to be
>> removed as well.
>>
>> - Wangda
>>
>>
>>
>> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
>> aagarwal@hortonworks.com> wrote:
>> Thanks for putting together this RC, Wangda.
>>
>> The guidance from Apache is to omit MD5s, specifically:
>> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>>
>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
>> wrote:
>>
>> Hi Gera,
>>
>> It's my bad, I thought only src/bin tarball is enough.
>>
>> I just uploaded all other things under artifact/ to
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>> Please let me know if you have any other comments.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
>> wrote:
>>
>>
>> Thanks, Wangda!
>>
>> There are many more artifacts in previous votes, e.g., see
>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>> site tarball is missing.
>>
>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>>
>>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>> UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
>> wrote:
>>
>>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>>
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via http://repository.apache.org at
>> https://repository.apache.org/content/repositories/
>> orgapachehadoop-1090/
>>
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services,
>> Support
>>
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>>
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>> as
>>
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>>
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Iñigo Goiri <el...@gmail.com>.
+1 (non binding)
* Deployed with 4 subclusters with HDFS Router-based federation.
* Executed DistCp across subclusters through the Router
* Checked documentation and tgz
On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
> > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > HI Arpit,
> >
> > I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com
> <ma...@hortonworks.com>> wrote:
> > Thanks Wangda, I see the shaded jars now.
> >
> > Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
> >
> > +1 (binding), modulo that remaining question.
> >
> > * Verified signatures
> > * Verified checksums for source and binary artefacts
> > * Sanity checked jars on r.a.o.
> > * Built from source
> > * Deployed to 3 node secure cluster with NameNode HA
> > * Verified HDFS web UIs
> > * Tried out HDFS shell commands
> > * Ran sample MapReduce jobs
> >
> > Thanks!
> >
> >
> > ----------------------------------------------------------------------
> > From: Wangda Tan <wheeleast@gmail.com <ma...@gmail.com>>
> > Date: Monday, April 2, 2018 at 9:25 PM
> > To: Arpit Agarwal <aagarwal@hortonworks.com <mailto:
> aagarwal@hortonworks.com>>
> > Cc: Gera Shegalov <gera05@gmail.com <ma...@gmail.com>>, Sunil G
> <sunilg@apache.org <ma...@apache.org>>, "
> yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>" <
> yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hdfs-dev
> <hdfs-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hadoop
> Common <common-dev@hadoop.apache.org <ma...@hadoop.apache.org>>,
> "mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>"
> <mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>>,
> Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>>
> > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> >
> > As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092 <https://repository.apache.org/content/repositories/
> orgapachehadoop-1092>
> >
> > Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
> >
> > Best,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com
> <ma...@gmail.com>> wrote:
> > Hi Arpit,
> >
> > Thanks for pointing out this.
> >
> > I just removed all .md5 files from artifacts. I found md5 checksums
> still exist in .mds files and I didn't remove them from .mds file because
> it is generated by create-release script and Apache guidance is "should
> not" instead of "must not". Please let me know if you think they need to be
> removed as well.
> >
> > - Wangda
> >
> >
> >
> > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com <ma...@hortonworks.com>> wrote:
> > Thanks for putting together this RC, Wangda.
> >
> > The guidance from Apache is to omit MD5s, specifically:
> > > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> >
> > https://www.apache.org/dev/release-distribution#sigs-and-sums <
> https://www.apache.org/dev/release-distribution#sigs-and-sums>
> >
> >
> >
> >
> > On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheeleast@gmail.com
> <ma...@gmail.com>> wrote:
> >
> > Hi Gera,
> >
> > It's my bad, I thought only src/bin tarball is enough.
> >
> > I just uploaded all other things under artifact/ to
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>
> >
> > Please let me know if you have any other comments.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:gera05@gmail.com
> <ma...@gmail.com>> wrote:
> >
> >
> > Thanks, Wangda!
> >
> > There are many more artifacts in previous votes, e.g., see
> > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ <
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/> . Among others the
> > site tarball is missing.
> >
> > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sunilg@apache.org
> <ma...@apache.org>> wrote:
> >
> >
> > Thanks Wangda for initiating the release.
> >
> > I tested this RC built from source file.
> >
> >
> > - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> > UI.
> > - Below feature sanity is done
> > - Application priority
> > - Application timeout
> > - Intra Queue preemption with priority based
> > - DS based affinity tests to verify placement constraints.
> > - Tested basic NodeLabel scenarios.
> > - Added couple of labels to few of nodes and behavior is coming
> > correct.
> > - Verified old UI and new YARN UI for labels.
> > - Submitted apps to labelled cluster and it works fine.
> > - Also performed few cli commands related to nodelabel.
> > - Test basic HA cases and seems correct.
> > - Tested new YARN UI . All pages are getting loaded correctly.
> >
> >
> > - Sunil
> >
> > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com
> <ma...@gmail.com>> wrote:
> >
> >
> > Hi folks,
> >
> > Thanks to the many who helped with this release since Dec 2017 [1].
> > We've
> >
> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1 <
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1>
> >
> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >
> > The maven artifacts are available via http://repository.apache.org <
> http://repository.apache.org/> at
> > https://repository.apache.org/content/repositories/ <
> https://repository.apache.org/content/repositories/>
> > orgapachehadoop-1090/
> >
> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >
> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > include the first class GPU/FPGA support on YARN, Native services,
> > Support
> >
> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > block replicas to be provided by an external storage system, etc.
> >
> > For 3.1.0 RC0 vote discussion, please see [3].
> >
> > We’d like to use this as a starting release for 3.1.x [1], depending on
> > how
> >
> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> > as
> >
> > the stable release.
> >
> > We have done testing with a pseudo cluster:
> > - Ran distributed job.
> > - GPU scheduling/isolation.
> > - Placement constraints (intra-application anti-affinity) by using
> > distributed shell.
> >
> > My +1 to start.
> >
> > Best,
> > Wangda/Vinod
> >
> > [1]
> >
> > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b <
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b>
> > c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org <
> http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> > BY
> >
> > fixVersion ASC
> > [3]
> >
> > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7 <
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7>
> > 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org <
> http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Iñigo Goiri <el...@gmail.com>.
+1 (non binding)
* Deployed with 4 subclusters with HDFS Router-based federation.
* Executed DistCp across subclusters through the Router
* Checked documentation and tgz
On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
> > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
> >
> > HI Arpit,
> >
> > I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com
> <ma...@hortonworks.com>> wrote:
> > Thanks Wangda, I see the shaded jars now.
> >
> > Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
> >
> > +1 (binding), modulo that remaining question.
> >
> > * Verified signatures
> > * Verified checksums for source and binary artefacts
> > * Sanity checked jars on r.a.o.
> > * Built from source
> > * Deployed to 3 node secure cluster with NameNode HA
> > * Verified HDFS web UIs
> > * Tried out HDFS shell commands
> > * Ran sample MapReduce jobs
> >
> > Thanks!
> >
> >
> > ----------------------------------------------------------------------
> > From: Wangda Tan <wheeleast@gmail.com <ma...@gmail.com>>
> > Date: Monday, April 2, 2018 at 9:25 PM
> > To: Arpit Agarwal <aagarwal@hortonworks.com <mailto:
> aagarwal@hortonworks.com>>
> > Cc: Gera Shegalov <gera05@gmail.com <ma...@gmail.com>>, Sunil G
> <sunilg@apache.org <ma...@apache.org>>, "
> yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>" <
> yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hdfs-dev
> <hdfs-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hadoop
> Common <common-dev@hadoop.apache.org <ma...@hadoop.apache.org>>,
> "mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>"
> <mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>>,
> Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>>
> > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> >
> > As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092 <https://repository.apache.org/content/repositories/
> orgapachehadoop-1092>
> >
> > Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
> >
> > Best,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com
> <ma...@gmail.com>> wrote:
> > Hi Arpit,
> >
> > Thanks for pointing out this.
> >
> > I just removed all .md5 files from artifacts. I found md5 checksums
> still exist in .mds files and I didn't remove them from .mds file because
> it is generated by create-release script and Apache guidance is "should
> not" instead of "must not". Please let me know if you think they need to be
> removed as well.
> >
> > - Wangda
> >
> >
> >
> > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com <ma...@hortonworks.com>> wrote:
> > Thanks for putting together this RC, Wangda.
> >
> > The guidance from Apache is to omit MD5s, specifically:
> > > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> >
> > https://www.apache.org/dev/release-distribution#sigs-and-sums <
> https://www.apache.org/dev/release-distribution#sigs-and-sums>
> >
> >
> >
> >
> > On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheeleast@gmail.com
> <ma...@gmail.com>> wrote:
> >
> > Hi Gera,
> >
> > It's my bad, I thought only src/bin tarball is enough.
> >
> > I just uploaded all other things under artifact/ to
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>
> >
> > Please let me know if you have any other comments.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:gera05@gmail.com
> <ma...@gmail.com>> wrote:
> >
> >
> > Thanks, Wangda!
> >
> > There are many more artifacts in previous votes, e.g., see
> > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ <
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/> . Among others the
> > site tarball is missing.
> >
> > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sunilg@apache.org
> <ma...@apache.org>> wrote:
> >
> >
> > Thanks Wangda for initiating the release.
> >
> > I tested this RC built from source file.
> >
> >
> > - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> > UI.
> > - Below feature sanity is done
> > - Application priority
> > - Application timeout
> > - Intra Queue preemption with priority based
> > - DS based affinity tests to verify placement constraints.
> > - Tested basic NodeLabel scenarios.
> > - Added couple of labels to few of nodes and behavior is coming
> > correct.
> > - Verified old UI and new YARN UI for labels.
> > - Submitted apps to labelled cluster and it works fine.
> > - Also performed few cli commands related to nodelabel.
> > - Test basic HA cases and seems correct.
> > - Tested new YARN UI . All pages are getting loaded correctly.
> >
> >
> > - Sunil
> >
> > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com
> <ma...@gmail.com>> wrote:
> >
> >
> > Hi folks,
> >
> > Thanks to the many who helped with this release since Dec 2017 [1].
> > We've
> >
> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1 <
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1>
> >
> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >
> > The maven artifacts are available via http://repository.apache.org <
> http://repository.apache.org/> at
> > https://repository.apache.org/content/repositories/ <
> https://repository.apache.org/content/repositories/>
> > orgapachehadoop-1090/
> >
> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >
> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > include the first class GPU/FPGA support on YARN, Native services,
> > Support
> >
> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > block replicas to be provided by an external storage system, etc.
> >
> > For 3.1.0 RC0 vote discussion, please see [3].
> >
> > We’d like to use this as a starting release for 3.1.x [1], depending on
> > how
> >
> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> > as
> >
> > the stable release.
> >
> > We have done testing with a pseudo cluster:
> > - Ran distributed job.
> > - GPU scheduling/isolation.
> > - Placement constraints (intra-application anti-affinity) by using
> > distributed shell.
> >
> > My +1 to start.
> >
> > Best,
> > Wangda/Vinod
> >
> > [1]
> >
> > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b <
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b>
> > c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org <
> http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> > BY
> >
> > fixVersion ASC
> > [3]
> >
> > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7 <
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7>
> > 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org <
> http://3cyarn-dev.hadoop.apache.org/>%3E
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Hi Vinod / Arpit,
I checked following versions:
- 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
Jars in maven repo [1] are *always* different from jars in the binary
tarball [2]: (I only checked hadoop-yarn-api-version.jar)
(Following numbers are sizes of the jar)
2.6.5:
- Jar in Maven: 1896185
- Jar in tarball: 1891485
2.7.5:
- Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
- Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
(Even size is same, md5 is different)
2.8.3:
- Jar in Maven: 2451433
- Jar in tarball: 2438975
2.9.0:
- Jar in Maven: 2791477
- Jar in tarball: 2777789
3.0.1:
- Jar in Maven: 2852604
- Jar in tarball: 2851373
I guess the differences come from our release process.
Thanks,
Wangda
[1] Maven jars are downloaded from
https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
<version>/hadoop-yarn-api-<version>.jar
[2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
>> Thanks Wangda, I see the shaded jars now.
>>
>> Are the repo jars required to be the same as the binary release? They
>> don’t match right now, probably they got rebuilt.
>>
>> +1 (binding), modulo that remaining question.
>>
>> * Verified signatures
>> * Verified checksums for source and binary artefacts
>> * Sanity checked jars on r.a.o.
>> * Built from source
>> * Deployed to 3 node secure cluster with NameNode HA
>> * Verified HDFS web UIs
>> * Tried out HDFS shell commands
>> * Ran sample MapReduce jobs
>>
>> Thanks!
>>
>>
>> ----------------------------------------------------------------------
>> From: Wangda Tan <wh...@gmail.com>
>> Date: Monday, April 2, 2018 at 9:25 PM
>> To: Arpit Agarwal <aa...@hortonworks.com>
>> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
>> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
>> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
>> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
>> Vinod Kumar Vavilapalli <vi...@apache.org>
>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>
>> As pointed by Arpit, the previously deployed shared jars are incorrect.
>> Just redeployed jars and staged. @Arpit, could you please check the updated
>> Maven repo? https://repository.apache.org/content/repositories/
>> orgapachehadoop-1092
>>
>> Since the jars inside binary tarballs are correct (
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
>> need roll another RC, just update Maven repo should be sufficient.
>>
>> Best,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
>> wrote:
>> Hi Arpit,
>>
>> Thanks for pointing out this.
>>
>> I just removed all .md5 files from artifacts. I found md5 checksums still
>> exist in .mds files and I didn't remove them from .mds file because it is
>> generated by create-release script and Apache guidance is "should not"
>> instead of "must not". Please let me know if you think they need to be
>> removed as well.
>>
>> - Wangda
>>
>>
>>
>> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
>> aagarwal@hortonworks.com> wrote:
>> Thanks for putting together this RC, Wangda.
>>
>> The guidance from Apache is to omit MD5s, specifically:
>> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>>
>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com>
>> wrote:
>>
>> Hi Gera,
>>
>> It's my bad, I thought only src/bin tarball is enough.
>>
>> I just uploaded all other things under artifact/ to
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>> Please let me know if you have any other comments.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
>> wrote:
>>
>>
>> Thanks, Wangda!
>>
>> There are many more artifacts in previous votes, e.g., see
>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>> site tarball is missing.
>>
>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>>
>>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>> UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
>> wrote:
>>
>>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>>
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via http://repository.apache.org at
>> https://repository.apache.org/content/repositories/
>> orgapachehadoop-1090/
>>
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services,
>> Support
>>
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>>
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>> as
>>
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>>
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
>> <http://3cyarn-dev.hadoop.apache.org/>%3E
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
We vote on the source code. The binaries are convenience artifacts.
This is what I would do - (a) Just replace both the maven jars as well as the binaries to be consistent and correct. And then (b) Give a couple more days for folks who tested on the binaries to reverify - I count one such clear vote as of now.
Thanks
+Vinod
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com <ma...@hortonworks.com>> wrote:
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wheeleast@gmail.com <ma...@gmail.com>>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagarwal@hortonworks.com <ma...@hortonworks.com>>
> Cc: Gera Shegalov <gera05@gmail.com <ma...@gmail.com>>, Sunil G <sunilg@apache.org <ma...@apache.org>>, "yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>" <yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hdfs-dev <hdfs-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hadoop Common <common-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, "mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>" <mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092 <https://repository.apache.org/content/repositories/orgapachehadoop-1092>
>
> Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>). I think we don't need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com <ma...@gmail.com>> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:aagarwal@hortonworks.com <ma...@hortonworks.com>> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums <https://www.apache.org/dev/release-distribution#sigs-and-sums>
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheeleast@gmail.com <ma...@gmail.com>> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:gera05@gmail.com <ma...@gmail.com>> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ <http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/> . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sunilg@apache.org <ma...@apache.org>> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com <ma...@gmail.com>> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1 <http://people.apache.org/~wangda/hadoop-3.1.0-RC1>
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org <http://repository.apache.org/> at
> https://repository.apache.org/content/repositories/ <https://repository.apache.org/content/repositories/>
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b <https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b>
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7 <https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7>
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
>
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
We vote on the source code. The binaries are convenience artifacts.
This is what I would do - (a) Just replace both the maven jars as well as the binaries to be consistent and correct. And then (b) Give a couple more days for folks who tested on the binaries to reverify - I count one such clear vote as of now.
Thanks
+Vinod
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com <ma...@hortonworks.com>> wrote:
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wheeleast@gmail.com <ma...@gmail.com>>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagarwal@hortonworks.com <ma...@hortonworks.com>>
> Cc: Gera Shegalov <gera05@gmail.com <ma...@gmail.com>>, Sunil G <sunilg@apache.org <ma...@apache.org>>, "yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>" <yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hdfs-dev <hdfs-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hadoop Common <common-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, "mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>" <mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092 <https://repository.apache.org/content/repositories/orgapachehadoop-1092>
>
> Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>). I think we don't need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com <ma...@gmail.com>> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:aagarwal@hortonworks.com <ma...@hortonworks.com>> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums <https://www.apache.org/dev/release-distribution#sigs-and-sums>
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheeleast@gmail.com <ma...@gmail.com>> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:gera05@gmail.com <ma...@gmail.com>> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ <http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/> . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sunilg@apache.org <ma...@apache.org>> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com <ma...@gmail.com>> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1 <http://people.apache.org/~wangda/hadoop-3.1.0-RC1>
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org <http://repository.apache.org/> at
> https://repository.apache.org/content/repositories/ <https://repository.apache.org/content/repositories/>
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b <https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b>
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7 <https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7>
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
>
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
We vote on the source code. The binaries are convenience artifacts.
This is what I would do - (a) Just replace both the maven jars as well as the binaries to be consistent and correct. And then (b) Give a couple more days for folks who tested on the binaries to reverify - I count one such clear vote as of now.
Thanks
+Vinod
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com <ma...@hortonworks.com>> wrote:
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wheeleast@gmail.com <ma...@gmail.com>>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagarwal@hortonworks.com <ma...@hortonworks.com>>
> Cc: Gera Shegalov <gera05@gmail.com <ma...@gmail.com>>, Sunil G <sunilg@apache.org <ma...@apache.org>>, "yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>" <yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hdfs-dev <hdfs-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hadoop Common <common-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, "mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>" <mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092 <https://repository.apache.org/content/repositories/orgapachehadoop-1092>
>
> Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>). I think we don't need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com <ma...@gmail.com>> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:aagarwal@hortonworks.com <ma...@hortonworks.com>> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums <https://www.apache.org/dev/release-distribution#sigs-and-sums>
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheeleast@gmail.com <ma...@gmail.com>> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:gera05@gmail.com <ma...@gmail.com>> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ <http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/> . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sunilg@apache.org <ma...@apache.org>> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com <ma...@gmail.com>> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1 <http://people.apache.org/~wangda/hadoop-3.1.0-RC1>
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org <http://repository.apache.org/> at
> https://repository.apache.org/content/repositories/ <https://repository.apache.org/content/repositories/>
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b <https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b>
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7 <https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7>
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
>
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
We vote on the source code. The binaries are convenience artifacts.
This is what I would do - (a) Just replace both the maven jars as well as the binaries to be consistent and correct. And then (b) Give a couple more days for folks who tested on the binaries to reverify - I count one such clear vote as of now.
Thanks
+Vinod
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wh...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com <ma...@hortonworks.com>> wrote:
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wheeleast@gmail.com <ma...@gmail.com>>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagarwal@hortonworks.com <ma...@hortonworks.com>>
> Cc: Gera Shegalov <gera05@gmail.com <ma...@gmail.com>>, Sunil G <sunilg@apache.org <ma...@apache.org>>, "yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>" <yarn-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hdfs-dev <hdfs-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Hadoop Common <common-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, "mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>" <mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>>, Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092 <https://repository.apache.org/content/repositories/orgapachehadoop-1092>
>
> Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>). I think we don't need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com <ma...@gmail.com>> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:aagarwal@hortonworks.com <ma...@hortonworks.com>> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums <https://www.apache.org/dev/release-distribution#sigs-and-sums>
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheeleast@gmail.com <ma...@gmail.com>> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:gera05@gmail.com <ma...@gmail.com>> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ <http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/> . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sunilg@apache.org <ma...@apache.org>> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com <ma...@gmail.com>> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1 <http://people.apache.org/~wangda/hadoop-3.1.0-RC1>
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org <http://repository.apache.org/> at
> https://repository.apache.org/content/repositories/ <https://repository.apache.org/content/repositories/>
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b <https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b>
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7 <https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7>
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
>
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
HI Arpit,
I think it won't match if we do rebuild. It should be fine as far as
they're signed, correct? I don't see any policy doesn't allow this.
Thanks,
Wangda
On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
wrote:
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aa...@hortonworks.com>
> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/
> repositories/orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
>
>
>
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
HI Arpit,
I think it won't match if we do rebuild. It should be fine as far as
they're signed, correct? I don't see any policy doesn't allow this.
Thanks,
Wangda
On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
wrote:
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aa...@hortonworks.com>
> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/
> repositories/orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
>
>
>
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
HI Arpit,
I think it won't match if we do rebuild. It should be fine as far as
they're signed, correct? I don't see any policy doesn't allow this.
Thanks,
Wangda
On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
wrote:
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aa...@hortonworks.com>
> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/
> repositories/orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
>
>
>
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
HI Arpit,
I think it won't match if we do rebuild. It should be fine as far as
they're signed, correct? I don't see any policy doesn't allow this.
Thanks,
Wangda
On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aa...@hortonworks.com>
wrote:
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <wh...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aa...@hortonworks.com>
> Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vi...@apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/
> repositories/orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> aagarwal@hortonworks.com> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
>
>
>
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Arpit Agarwal <aa...@hortonworks.com>.
Thanks Wangda, I see the shaded jars now.
Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
+1 (binding), modulo that remaining question.
* Verified signatures
* Verified checksums for source and binary artefacts
* Sanity checked jars on r.a.o.
* Built from source
* Deployed to 3 node secure cluster with NameNode HA
* Verified HDFS web UIs
* Tried out HDFS shell commands
* Ran sample MapReduce jobs
Thanks!
----------------------------------------------------------------------
From: Wangda Tan <wh...@gmail.com>
Date: Monday, April 2, 2018 at 9:25 PM
To: Arpit Agarwal <aa...@hortonworks.com>
Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <hd...@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@apache.org>
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092
Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll another RC, just update Maven repo should be sufficient.
Best,
Wangda
On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com> wrote:
Hi Arpit,
Thanks for pointing out this.
I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
- Wangda
On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <ma...@hortonworks.com> wrote:
Thanks for putting together this RC, Wangda.
The guidance from Apache is to omit MD5s, specifically:
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
https://www.apache.org/dev/release-distribution#sigs-and-sums
On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com> wrote:
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via http://repository.apache.org at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on
how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Arpit Agarwal <aa...@hortonworks.com>.
Thanks Wangda, I see the shaded jars now.
Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
+1 (binding), modulo that remaining question.
* Verified signatures
* Verified checksums for source and binary artefacts
* Sanity checked jars on r.a.o.
* Built from source
* Deployed to 3 node secure cluster with NameNode HA
* Verified HDFS web UIs
* Tried out HDFS shell commands
* Ran sample MapReduce jobs
Thanks!
----------------------------------------------------------------------
From: Wangda Tan <wh...@gmail.com>
Date: Monday, April 2, 2018 at 9:25 PM
To: Arpit Agarwal <aa...@hortonworks.com>
Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <hd...@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@apache.org>
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092
Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll another RC, just update Maven repo should be sufficient.
Best,
Wangda
On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com> wrote:
Hi Arpit,
Thanks for pointing out this.
I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
- Wangda
On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <ma...@hortonworks.com> wrote:
Thanks for putting together this RC, Wangda.
The guidance from Apache is to omit MD5s, specifically:
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
https://www.apache.org/dev/release-distribution#sigs-and-sums
On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com> wrote:
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via http://repository.apache.org at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on
how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
RE: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Takanobu Asanuma <ta...@yahoo-corp.jp>.
Thanks for your efforts, Wangda. Sorry for participating late.
+1 (non-binding)
- Verified the checksum of the tarball
- Succeeded "mvn clean package -Pdist,native -Dtar -DskipTests"
- Started docker hadoop cluster with 1 master and 5 slaves
- Verified TeraGen/TeraSort
- Verified some erasure coding operations
- Seems the shaded jars (hadoop-client-runtime, hadoop-client-api, hadoop-client-minicluster) have correct sizes in orgapachehadoop-1092.
-Takanobu
-----Original Message-----
From: Wangda Tan [mailto:wheeleast@gmail.com]
Sent: Tuesday, April 03, 2018 1:25 PM
To: Arpit Agarwal <aa...@hortonworks.com>
Cc: Gera Shegalov <ge...@gmail.com>; Sunil G <su...@apache.org>; yarn-dev@hadoop.apache.org; Hdfs-dev <hd...@hadoop.apache.org>; Hadoop Common <co...@hadoop.apache.org>; mapreduce-dev@hadoop.apache.org; Vinod Kumar Vavilapalli <vi...@apache.org>
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo?
https://repository.apache.org/content/repositories/orgapachehadoop-1092
Since the jars inside binary tarballs are correct ( http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll another RC, just update Maven repo should be sufficient.
Best,
Wangda
On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <wh...@gmail.com> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums
> still exist in .mds files and I didn't remove them from .mds file
> because it is generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal
> <aa...@hortonworks.com>
> wrote:
>
>> Thanks for putting together this RC, Wangda.
>>
>> The guidance from Apache is to omit MD5s, specifically:
>> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi Gera,
>>
>> It's my bad, I thought only src/bin tarball is enough.
>>
>> I just uploaded all other things under artifact/ to
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>> Please let me know if you have any other comments.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
>>
>> Thanks, Wangda!
>>
>> There are many more artifacts in previous votes, e.g., see
>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others
>> the site tarball is missing.
>>
>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
>>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old
>> RM UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1].
>>
>> We've
>>
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/
>>
>> orgapachehadoop-1090/
>>
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable
>> additions include the first class GPU/FPGA support on YARN, Native
>> services,
>>
>> Support
>>
>> rich placement constraints in YARN, S3-related enhancements, allow
>> HDFS block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending
>> on
>>
>> how
>>
>> it goes, get it stabilized and potentially use a 3.1.1 in several
>> weeks
>>
>> as
>>
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>>
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>> (3.1.0) AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status =
>> Resolved ORDER
>>
>> BY
>>
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>>
>>
>>
>>
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Arpit Agarwal <aa...@hortonworks.com>.
Thanks Wangda, I see the shaded jars now.
Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
+1 (binding), modulo that remaining question.
* Verified signatures
* Verified checksums for source and binary artefacts
* Sanity checked jars on r.a.o.
* Built from source
* Deployed to 3 node secure cluster with NameNode HA
* Verified HDFS web UIs
* Tried out HDFS shell commands
* Ran sample MapReduce jobs
Thanks!
----------------------------------------------------------------------
From: Wangda Tan <wh...@gmail.com>
Date: Monday, April 2, 2018 at 9:25 PM
To: Arpit Agarwal <aa...@hortonworks.com>
Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <hd...@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@apache.org>
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092
Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll another RC, just update Maven repo should be sufficient.
Best,
Wangda
On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com> wrote:
Hi Arpit,
Thanks for pointing out this.
I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
- Wangda
On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <ma...@hortonworks.com> wrote:
Thanks for putting together this RC, Wangda.
The guidance from Apache is to omit MD5s, specifically:
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
https://www.apache.org/dev/release-distribution#sigs-and-sums
On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com> wrote:
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via http://repository.apache.org at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on
how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Arpit Agarwal <aa...@hortonworks.com>.
Thanks Wangda, I see the shaded jars now.
Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
+1 (binding), modulo that remaining question.
* Verified signatures
* Verified checksums for source and binary artefacts
* Sanity checked jars on r.a.o.
* Built from source
* Deployed to 3 node secure cluster with NameNode HA
* Verified HDFS web UIs
* Tried out HDFS shell commands
* Ran sample MapReduce jobs
Thanks!
----------------------------------------------------------------------
From: Wangda Tan <wh...@gmail.com>
Date: Monday, April 2, 2018 at 9:25 PM
To: Arpit Agarwal <aa...@hortonworks.com>
Cc: Gera Shegalov <ge...@gmail.com>, Sunil G <su...@apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, Hdfs-dev <hd...@hadoop.apache.org>, Hadoop Common <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@apache.org>
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092
Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll another RC, just update Maven repo should be sufficient.
Best,
Wangda
On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <ma...@gmail.com> wrote:
Hi Arpit,
Thanks for pointing out this.
I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
- Wangda
On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <ma...@hortonworks.com> wrote:
Thanks for putting together this RC, Wangda.
The guidance from Apache is to omit MD5s, specifically:
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
https://www.apache.org/dev/release-distribution#sigs-and-sums
On Apr 2, 2018, at 7:03 AM, Wangda Tan <ma...@gmail.com> wrote:
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ma...@gmail.com> wrote:
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <ma...@apache.org> wrote:
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <ma...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via http://repository.apache.org at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on
how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated
Maven repo?
https://repository.apache.org/content/repositories/orgapachehadoop-1092
Since the jars inside binary tarballs are correct (
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need
roll another RC, just update Maven repo should be sufficient.
Best,
Wangda
On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <wh...@gmail.com> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
>> Thanks for putting together this RC, Wangda.
>>
>> The guidance from Apache is to omit MD5s, specifically:
>> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi Gera,
>>
>> It's my bad, I thought only src/bin tarball is enough.
>>
>> I just uploaded all other things under artifact/ to
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>> Please let me know if you have any other comments.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
>>
>> Thanks, Wangda!
>>
>> There are many more artifacts in previous votes, e.g., see
>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>> site tarball is missing.
>>
>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
>>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>> UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1].
>>
>> We've
>>
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/
>>
>> orgapachehadoop-1090/
>>
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services,
>>
>> Support
>>
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>>
>> how
>>
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>>
>> as
>>
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>>
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>>
>> BY
>>
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>>
>>
>>
>>
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated
Maven repo?
https://repository.apache.org/content/repositories/orgapachehadoop-1092
Since the jars inside binary tarballs are correct (
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need
roll another RC, just update Maven repo should be sufficient.
Best,
Wangda
On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <wh...@gmail.com> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
>> Thanks for putting together this RC, Wangda.
>>
>> The guidance from Apache is to omit MD5s, specifically:
>> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi Gera,
>>
>> It's my bad, I thought only src/bin tarball is enough.
>>
>> I just uploaded all other things under artifact/ to
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>> Please let me know if you have any other comments.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
>>
>> Thanks, Wangda!
>>
>> There are many more artifacts in previous votes, e.g., see
>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>> site tarball is missing.
>>
>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
>>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>> UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1].
>>
>> We've
>>
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/
>>
>> orgapachehadoop-1090/
>>
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services,
>>
>> Support
>>
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>>
>> how
>>
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>>
>> as
>>
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>>
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>>
>> BY
>>
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>>
>>
>>
>>
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated
Maven repo?
https://repository.apache.org/content/repositories/orgapachehadoop-1092
Since the jars inside binary tarballs are correct (
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need
roll another RC, just update Maven repo should be sufficient.
Best,
Wangda
On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <wh...@gmail.com> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
>> Thanks for putting together this RC, Wangda.
>>
>> The guidance from Apache is to omit MD5s, specifically:
>> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi Gera,
>>
>> It's my bad, I thought only src/bin tarball is enough.
>>
>> I just uploaded all other things under artifact/ to
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>> Please let me know if you have any other comments.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
>>
>> Thanks, Wangda!
>>
>> There are many more artifacts in previous votes, e.g., see
>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>> site tarball is missing.
>>
>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
>>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>> UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1].
>>
>> We've
>>
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/
>>
>> orgapachehadoop-1090/
>>
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services,
>>
>> Support
>>
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>>
>> how
>>
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>>
>> as
>>
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>>
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>>
>> BY
>>
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>>
>>
>>
>>
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated
Maven repo?
https://repository.apache.org/content/repositories/orgapachehadoop-1092
Since the jars inside binary tarballs are correct (
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need
roll another RC, just update Maven repo should be sufficient.
Best,
Wangda
On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <wh...@gmail.com> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <aa...@hortonworks.com>
> wrote:
>
>> Thanks for putting together this RC, Wangda.
>>
>> The guidance from Apache is to omit MD5s, specifically:
>> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi Gera,
>>
>> It's my bad, I thought only src/bin tarball is enough.
>>
>> I just uploaded all other things under artifact/ to
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>> Please let me know if you have any other comments.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
>>
>> Thanks, Wangda!
>>
>> There are many more artifacts in previous votes, e.g., see
>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
>> site tarball is missing.
>>
>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
>>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>> UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1].
>>
>> We've
>>
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/
>>
>> orgapachehadoop-1090/
>>
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services,
>>
>> Support
>>
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>>
>> how
>>
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>>
>> as
>>
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>>
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>>
>> BY
>>
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>>
>>
>>
>>
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Hi Arpit,
Thanks for pointing out this.
I just removed all .md5 files from artifacts. I found md5 checksums still
exist in .mds files and I didn't remove them from .mds file because it is
generated by create-release script and Apache guidance is "should not"
instead of "must not". Please let me know if you think they need to be
removed as well.
- Wangda
On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <aa...@hortonworks.com>
wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
>
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/
>
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
>
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
>
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Hi Arpit,
Thanks for pointing out this.
I just removed all .md5 files from artifacts. I found md5 checksums still
exist in .mds files and I didn't remove them from .mds file because it is
generated by create-release script and Apache guidance is "should not"
instead of "must not". Please let me know if you think they need to be
removed as well.
- Wangda
On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <aa...@hortonworks.com>
wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
>
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/
>
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
>
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
>
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Hi Arpit,
Thanks for pointing out this.
I just removed all .md5 files from artifacts. I found md5 checksums still
exist in .mds files and I didn't remove them from .mds file because it is
generated by create-release script and Apache guidance is "should not"
instead of "must not". Please let me know if you think they need to be
removed as well.
- Wangda
On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <aa...@hortonworks.com>
wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
>
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/
>
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
>
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
>
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Hi Arpit,
Thanks for pointing out this.
I just removed all .md5 files from artifacts. I found md5 checksums still
exist in .mds files and I didn't remove them from .mds file because it is
generated by create-release script and Apache guidance is "should not"
instead of "must not". Please let me know if you think they need to be
removed as well.
- Wangda
On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <aa...@hortonworks.com>
wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <wh...@gmail.com> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
>
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/
>
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
>
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
>
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Arpit Agarwal <aa...@hortonworks.com>.
Thanks for putting together this RC, Wangda.
The guidance from Apache is to omit MD5s, specifically:
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
https://www.apache.org/dev/release-distribution#sigs-and-sums
On Apr 2, 2018, at 7:03 AM, Wangda Tan <wh...@gmail.com>> wrote:
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on
how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Arpit Agarwal <aa...@hortonworks.com>.
Thanks for putting together this RC, Wangda.
The guidance from Apache is to omit MD5s, specifically:
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
https://www.apache.org/dev/release-distribution#sigs-and-sums
On Apr 2, 2018, at 7:03 AM, Wangda Tan <wh...@gmail.com>> wrote:
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on
how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Arpit Agarwal <aa...@hortonworks.com>.
Thanks for putting together this RC, Wangda.
The guidance from Apache is to omit MD5s, specifically:
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
https://www.apache.org/dev/release-distribution#sigs-and-sums
On Apr 2, 2018, at 7:03 AM, Wangda Tan <wh...@gmail.com>> wrote:
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on
how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Arpit Agarwal <aa...@hortonworks.com>.
Thanks for putting together this RC, Wangda.
The guidance from Apache is to omit MD5s, specifically:
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
https://www.apache.org/dev/release-distribution#sigs-and-sums
On Apr 2, 2018, at 7:03 AM, Wangda Tan <wh...@gmail.com>> wrote:
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on
how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>> UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>>
>> > Hi folks,
>> >
>> > Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>> >
>> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>> >
>> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>> >
>> > The maven artifacts are available via repository.apache.org at
>> > https://repository.apache.org/content/repositories/
>> orgapachehadoop-1090/
>> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>> >
>> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> > include the first class GPU/FPGA support on YARN, Native services,
>> Support
>> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> > block replicas to be provided by an external storage system, etc.
>> >
>> > For 3.1.0 RC0 vote discussion, please see [3].
>> >
>> > We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>> as
>> > the stable release.
>> >
>> > We have done testing with a pseudo cluster:
>> > - Ran distributed job.
>> > - GPU scheduling/isolation.
>> > - Placement constraints (intra-application anti-affinity) by using
>> > distributed shell.
>> >
>> > My +1 to start.
>> >
>> > Best,
>> > Wangda/Vinod
>> >
>> > [1]
>> >
>> > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>> > fixVersion ASC
>> > [3]
>> >
>> > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>> UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>>
>> > Hi folks,
>> >
>> > Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>> >
>> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>> >
>> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>> >
>> > The maven artifacts are available via repository.apache.org at
>> > https://repository.apache.org/content/repositories/
>> orgapachehadoop-1090/
>> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>> >
>> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> > include the first class GPU/FPGA support on YARN, Native services,
>> Support
>> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> > block replicas to be provided by an external storage system, etc.
>> >
>> > For 3.1.0 RC0 vote discussion, please see [3].
>> >
>> > We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>> as
>> > the stable release.
>> >
>> > We have done testing with a pseudo cluster:
>> > - Ran distributed job.
>> > - GPU scheduling/isolation.
>> > - Placement constraints (intra-application anti-affinity) by using
>> > distributed shell.
>> >
>> > My +1 to start.
>> >
>> > Best,
>> > Wangda/Vinod
>> >
>> > [1]
>> >
>> > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>> > fixVersion ASC
>> > [3]
>> >
>> > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>> UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>>
>> > Hi folks,
>> >
>> > Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>> >
>> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>> >
>> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>> >
>> > The maven artifacts are available via repository.apache.org at
>> > https://repository.apache.org/content/repositories/
>> orgapachehadoop-1090/
>> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>> >
>> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> > include the first class GPU/FPGA support on YARN, Native services,
>> Support
>> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> > block replicas to be provided by an external storage system, etc.
>> >
>> > For 3.1.0 RC0 vote discussion, please see [3].
>> >
>> > We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>> as
>> > the stable release.
>> >
>> > We have done testing with a pseudo cluster:
>> > - Ran distributed job.
>> > - GPU scheduling/isolation.
>> > - Placement constraints (intra-application anti-affinity) by using
>> > distributed shell.
>> >
>> > My +1 to start.
>> >
>> > Best,
>> > Wangda/Vinod
>> >
>> > [1]
>> >
>> > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>> > fixVersion ASC
>> > [3]
>> >
>> > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Hi Gera,
It's my bad, I thought only src/bin tarball is enough.
I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
Please let me know if you have any other comments.
Thanks,
Wangda
On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ge...@gmail.com> wrote:
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>> UI.
>> - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>> - Tested basic NodeLabel scenarios.
>> - Added couple of labels to few of nodes and behavior is coming
>> correct.
>> - Verified old UI and new YARN UI for labels.
>> - Submitted apps to labelled cluster and it works fine.
>> - Also performed few cli commands related to nodelabel.
>> - Test basic HA cases and seems correct.
>> - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>>
>> > Hi folks,
>> >
>> > Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>> >
>> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>> >
>> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>> >
>> > The maven artifacts are available via repository.apache.org at
>> > https://repository.apache.org/content/repositories/
>> orgapachehadoop-1090/
>> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>> >
>> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> > include the first class GPU/FPGA support on YARN, Native services,
>> Support
>> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> > block replicas to be provided by an external storage system, etc.
>> >
>> > For 3.1.0 RC0 vote discussion, please see [3].
>> >
>> > We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>> as
>> > the stable release.
>> >
>> > We have done testing with a pseudo cluster:
>> > - Ran distributed job.
>> > - GPU scheduling/isolation.
>> > - Placement constraints (intra-application anti-affinity) by using
>> > distributed shell.
>> >
>> > My +1 to start.
>> >
>> > Best,
>> > Wangda/Vinod
>> >
>> > [1]
>> >
>> > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>> > fixVersion ASC
>> > [3]
>> >
>> > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>> >
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Gera Shegalov <ge...@gmail.com>.
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>
> > Hi folks,
> >
> > Thanks to the many who helped with this release since Dec 2017 [1]. We've
> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >
> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >
> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > include the first class GPU/FPGA support on YARN, Native services,
> Support
> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > block replicas to be provided by an external storage system, etc.
> >
> > For 3.1.0 RC0 vote discussion, please see [3].
> >
> > We’d like to use this as a starting release for 3.1.x [1], depending on
> how
> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
> > the stable release.
> >
> > We have done testing with a pseudo cluster:
> > - Ran distributed job.
> > - GPU scheduling/isolation.
> > - Placement constraints (intra-application anti-affinity) by using
> > distributed shell.
> >
> > My +1 to start.
> >
> > Best,
> > Wangda/Vinod
> >
> > [1]
> >
> >
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> > fixVersion ASC
> > [3]
> >
> >
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Gera Shegalov <ge...@gmail.com>.
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>
> > Hi folks,
> >
> > Thanks to the many who helped with this release since Dec 2017 [1]. We've
> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >
> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >
> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > include the first class GPU/FPGA support on YARN, Native services,
> Support
> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > block replicas to be provided by an external storage system, etc.
> >
> > For 3.1.0 RC0 vote discussion, please see [3].
> >
> > We’d like to use this as a starting release for 3.1.x [1], depending on
> how
> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
> > the stable release.
> >
> > We have done testing with a pseudo cluster:
> > - Ran distributed job.
> > - GPU scheduling/isolation.
> > - Placement constraints (intra-application anti-affinity) by using
> > distributed shell.
> >
> > My +1 to start.
> >
> > Best,
> > Wangda/Vinod
> >
> > [1]
> >
> >
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> > fixVersion ASC
> > [3]
> >
> >
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Sunil G <su...@apache.org>.
+1 (binding)
On Mon 2 Apr, 2018, 12:24 Sunil G, <su...@apache.org> wrote:
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1]. We've
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services, Support
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Sunil G <su...@apache.org>.
+1 (binding)
On Mon 2 Apr, 2018, 12:24 Sunil G, <su...@apache.org> wrote:
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1]. We've
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services, Support
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Gera Shegalov <ge...@gmail.com>.
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>
> > Hi folks,
> >
> > Thanks to the many who helped with this release since Dec 2017 [1]. We've
> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >
> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >
> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > include the first class GPU/FPGA support on YARN, Native services,
> Support
> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > block replicas to be provided by an external storage system, etc.
> >
> > For 3.1.0 RC0 vote discussion, please see [3].
> >
> > We’d like to use this as a starting release for 3.1.x [1], depending on
> how
> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
> > the stable release.
> >
> > We have done testing with a pseudo cluster:
> > - Ran distributed job.
> > - GPU scheduling/isolation.
> > - Placement constraints (intra-application anti-affinity) by using
> > distributed shell.
> >
> > My +1 to start.
> >
> > Best,
> > Wangda/Vinod
> >
> > [1]
> >
> >
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> > fixVersion ASC
> > [3]
> >
> >
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Sunil G <su...@apache.org>.
+1 (binding)
On Mon 2 Apr, 2018, 12:24 Sunil G, <su...@apache.org> wrote:
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1]. We've
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services, Support
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Gera Shegalov <ge...@gmail.com>.
Thanks, Wangda!
There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the
site tarball is missing.
On Sun, Apr 1, 2018 at 11:54 PM Sunil G <su...@apache.org> wrote:
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>
> > Hi folks,
> >
> > Thanks to the many who helped with this release since Dec 2017 [1]. We've
> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >
> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >
> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > include the first class GPU/FPGA support on YARN, Native services,
> Support
> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > block replicas to be provided by an external storage system, etc.
> >
> > For 3.1.0 RC0 vote discussion, please see [3].
> >
> > We’d like to use this as a starting release for 3.1.x [1], depending on
> how
> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
> > the stable release.
> >
> > We have done testing with a pseudo cluster:
> > - Ran distributed job.
> > - GPU scheduling/isolation.
> > - Placement constraints (intra-application anti-affinity) by using
> > distributed shell.
> >
> > My +1 to start.
> >
> > Best,
> > Wangda/Vinod
> >
> > [1]
> >
> >
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> > fixVersion ASC
> > [3]
> >
> >
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Sunil G <su...@apache.org>.
+1 (binding)
On Mon 2 Apr, 2018, 12:24 Sunil G, <su...@apache.org> wrote:
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
> - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
> - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
> - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
> - Test basic HA cases and seems correct.
> - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1]. We've
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services, Support
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Sunil G <su...@apache.org>.
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Sunil G <su...@apache.org>.
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Konstantinos Karanasos <kk...@gmail.com>.
+1 (binding)
Thanks for putting this together, Wangda.
I deployed a 7-node hadoop cluster and:
- ran various MR jobs
- ran the same jobs with a mix of opportunistic containers via centralized
scheduling
- ran the jobs with opportunistic containers and distributed scheduling
- ran some jobs with various placement constraints.
All worked as expected.
Thanks,
Konstantinos
On Thu, Mar 29, 2018 at 9:15 PM, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Eric Payne <er...@yahoo.com.INVALID>.
+1 (binding)
Thanks Wangda for doing the work to produce this release.
I did the following to test the release:
- Built from source
- Installed on 6-node pseudo cluster
- Interacted with RM CLI and GUI
- Tested streaming jobs
- Tested yarn distributed shell jobs
- Tested Max AM Resource Percent
- Tested simple inter-queue preemption
- Tested priority first intra-queue preemption
- Tested userlimit first intra-queue preemption
Thanks,Eric Payne
===========================================================
On Thursday, March 29, 2018, 11:15:51 PM CDT, Wangda Tan <wh...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Rohith Sharma K S <ro...@apache.org>.
+1 (binding)
* Downloaded source, built from source with -Dhbase.profile=2.0.
* Installed RM HA cluster integrated with ATSv2. Installed HBase-2.0-beta1
for ATSv2 back end. Scheduler is configured with 2 level queue hierarchy.
* Ran sample jobs such as MR/Distributed shell and verified for
** ATSv2 Timeline Reader REST API's. Validated for data published in ATSv2.
** YARN UI2 accessed for Flow Activity pages. Navigated inside flow
activity page for all other info.
** YARN old UI also verified for all the pages
** RM HA/Restart/work-preserving-restart are tested while running a job.
** NM restart scenarios are verified.
** Application timeout and application priority feature is tested.
Thanks & Regards
Rohith Sharma K S
On 30 March 2018 at 09:45, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Weiwei Yang <ch...@hotmail.com>.
Hi Wangda
+1 (non-binding)
- Smoke tests with teragen/terasort/DS jobs
- Various of metrics, UI displays validation, compatible tests
- Tested GPU resource-type
- Verified RM fail-over, app-recovery
- Verified 2 threads async-scheduling
- Enabled placement constraints, tested affinity/anti-affinity allocations
- SLS tests
--
Weiwei
On 2 Apr 2018, 1:13 PM +0800, Brahma Reddy Battula <br...@gmail.com>, wrote:
Wangda thanks for driving this.
+1(binding)
--Built from source
--Installed HA cluster
--Verified Basic Shell commands
--Ran Sample Jobs
--Browsed the UI's.
On Fri, Mar 30, 2018 at 9:45 AM, Wangda Tan <wh...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Weiwei Yang <ch...@hotmail.com>.
Hi Wangda
+1 (non-binding)
- Smoke tests with teragen/terasort/DS jobs
- Various of metrics, UI displays validation, compatible tests
- Tested GPU resource-type
- Verified RM fail-over, app-recovery
- Verified 2 threads async-scheduling
- Enabled placement constraints, tested affinity/anti-affinity allocations
- SLS tests
--
Weiwei
On 2 Apr 2018, 1:13 PM +0800, Brahma Reddy Battula <br...@gmail.com>, wrote:
Wangda thanks for driving this.
+1(binding)
--Built from source
--Installed HA cluster
--Verified Basic Shell commands
--Ran Sample Jobs
--Browsed the UI's.
On Fri, Mar 30, 2018 at 9:45 AM, Wangda Tan <wh...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Weiwei Yang <ch...@hotmail.com>.
Hi Wangda
+1 (non-binding)
- Smoke tests with teragen/terasort/DS jobs
- Various of metrics, UI displays validation, compatible tests
- Tested GPU resource-type
- Verified RM fail-over, app-recovery
- Verified 2 threads async-scheduling
- Enabled placement constraints, tested affinity/anti-affinity allocations
- SLS tests
--
Weiwei
On 2 Apr 2018, 1:13 PM +0800, Brahma Reddy Battula <br...@gmail.com>, wrote:
Wangda thanks for driving this.
+1(binding)
--Built from source
--Installed HA cluster
--Verified Basic Shell commands
--Ran Sample Jobs
--Browsed the UI's.
On Fri, Mar 30, 2018 at 9:45 AM, Wangda Tan <wh...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Weiwei Yang <ch...@hotmail.com>.
Hi Wangda
+1 (non-binding)
- Smoke tests with teragen/terasort/DS jobs
- Various of metrics, UI displays validation, compatible tests
- Tested GPU resource-type
- Verified RM fail-over, app-recovery
- Verified 2 threads async-scheduling
- Enabled placement constraints, tested affinity/anti-affinity allocations
- SLS tests
--
Weiwei
On 2 Apr 2018, 1:13 PM +0800, Brahma Reddy Battula <br...@gmail.com>, wrote:
Wangda thanks for driving this.
+1(binding)
--Built from source
--Installed HA cluster
--Verified Basic Shell commands
--Ran Sample Jobs
--Browsed the UI's.
On Fri, Mar 30, 2018 at 9:45 AM, Wangda Tan <wh...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Brahma Reddy Battula <br...@gmail.com>.
Wangda thanks for driving this.
+1(binding)
--Built from source
--Installed HA cluster
--Verified Basic Shell commands
--Ran Sample Jobs
--Browsed the UI's.
On Fri, Mar 30, 2018 at 9:45 AM, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Yufei Gu <fl...@gmail.com>.
Thanks Wangda for working on this!
+1 (non-binding)
- Downloaded the binary tarball and verified the checksum.
- Started a pseudo cluster inside one docker container
- Run Resource Manager with Fair Scheduler
- Verified distributed shell
- Verified mapreduce pi job
- Sanity checked RM WebUI
Best,
Yufei
On Thu, Mar 29, 2018 at 9:15 PM, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Brahma Reddy Battula <br...@gmail.com>.
Wangda thanks for driving this.
+1(binding)
--Built from source
--Installed HA cluster
--Verified Basic Shell commands
--Ran Sample Jobs
--Browsed the UI's.
On Fri, Mar 30, 2018 at 9:45 AM, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Brahma Reddy Battula <br...@gmail.com>.
Wangda thanks for driving this.
+1(binding)
--Built from source
--Installed HA cluster
--Verified Basic Shell commands
--Ran Sample Jobs
--Browsed the UI's.
On Fri, Mar 30, 2018 at 9:45 AM, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Yufei Gu <fl...@gmail.com>.
Thanks Wangda for working on this!
+1 (non-binding)
- Downloaded the binary tarball and verified the checksum.
- Started a pseudo cluster inside one docker container
- Run Resource Manager with Fair Scheduler
- Verified distributed shell
- Verified mapreduce pi job
- Sanity checked RM WebUI
Best,
Yufei
On Thu, Mar 29, 2018 at 9:15 PM, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Sunil G <su...@apache.org>.
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Thanks @Konstantin for reporting this issue, I will post comments on the
JIRA (HADOOP-15205)
- Wangda
On Tue, Apr 10, 2018 at 12:08 PM, Konstantin Shvachko <sh...@gmail.com>
wrote:
> A note to release managers. As discussed in
> https://issues.apache.org/jira/browse/HADOOP-15205
> We are producing release artifacts without sources jars. See e.g.
> https://repository.apache.org/content/repositories/releases/
> org/apache/hadoop/hadoop-common/3.1.0/
> I believe this has something to do with maven deployment stage,
> potentially maven-source-plugin.
> This is similar for all releases now, and I believe it should be fixed.
>
> Thanks,
> --Konstantin
>
> On Fri, Apr 6, 2018 at 2:01 PM, Wangda Tan <wh...@gmail.com> wrote:
>
>> Thanks guys for the additional votes! I just sent out announcement email.
>>
>> Best,
>> Wangda
>>
>> On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵 <ju...@apache.org> wrote:
>>
>> > Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
>> > based on:
>> >
>> > - Verified signatures
>> >
>> > - Verified checksums for source and binary artifacts
>> >
>> > - Built from source
>> >
>> > - Deployed a single node cluster
>> >
>> > - Verified web UIs, include Namenode, RM, etc.
>> >
>> > * Tried shell commands of HDFS and YARN
>> >
>> > * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
>> >
>> >
>> > Thanks,
>> >
>> >
>> > Junping
>> >
>> >
>> >
>> > Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
>> >
>> >> Hi folks,
>> >>
>> >> Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>> >> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>> >>
>> >> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>> >>
>> >> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> >> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>> >>
>> >> The maven artifacts are available via repository.apache.org at
>> >> https://repository.apache.org/content/repositories/orgapache
>> hadoop-1090/
>> >> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>> >>
>> >> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> >> include the first class GPU/FPGA support on YARN, Native services,
>> Support
>> >> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> >> block replicas to be provided by an external storage system, etc.
>> >>
>> >> For 3.1.0 RC0 vote discussion, please see [3].
>> >>
>> >> We’d like to use this as a starting release for 3.1.x [1], depending on
>> >> how
>> >> it goes, get it stabilized and potentially use a 3.1.1 in several
>> weeks as
>> >> the stable release.
>> >>
>> >> We have done testing with a pseudo cluster:
>> >> - Ran distributed job.
>> >> - GPU scheduling/isolation.
>> >> - Placement constraints (intra-application anti-affinity) by using
>> >> distributed shell.
>> >>
>> >> My +1 to start.
>> >>
>> >> Best,
>> >> Wangda/Vinod
>> >>
>> >> [1]
>> >> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> >> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> >> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>> (3.1.0)
>> >> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>> >> fixVersion ASC
>> >> [3]
>> >> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> >> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>> >>
>> >
>>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Thanks @Konstantin for reporting this issue, I will post comments on the
JIRA (HADOOP-15205)
- Wangda
On Tue, Apr 10, 2018 at 12:08 PM, Konstantin Shvachko <sh...@gmail.com>
wrote:
> A note to release managers. As discussed in
> https://issues.apache.org/jira/browse/HADOOP-15205
> We are producing release artifacts without sources jars. See e.g.
> https://repository.apache.org/content/repositories/releases/
> org/apache/hadoop/hadoop-common/3.1.0/
> I believe this has something to do with maven deployment stage,
> potentially maven-source-plugin.
> This is similar for all releases now, and I believe it should be fixed.
>
> Thanks,
> --Konstantin
>
> On Fri, Apr 6, 2018 at 2:01 PM, Wangda Tan <wh...@gmail.com> wrote:
>
>> Thanks guys for the additional votes! I just sent out announcement email.
>>
>> Best,
>> Wangda
>>
>> On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵 <ju...@apache.org> wrote:
>>
>> > Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
>> > based on:
>> >
>> > - Verified signatures
>> >
>> > - Verified checksums for source and binary artifacts
>> >
>> > - Built from source
>> >
>> > - Deployed a single node cluster
>> >
>> > - Verified web UIs, include Namenode, RM, etc.
>> >
>> > * Tried shell commands of HDFS and YARN
>> >
>> > * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
>> >
>> >
>> > Thanks,
>> >
>> >
>> > Junping
>> >
>> >
>> >
>> > Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
>> >
>> >> Hi folks,
>> >>
>> >> Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>> >> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>> >>
>> >> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>> >>
>> >> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> >> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>> >>
>> >> The maven artifacts are available via repository.apache.org at
>> >> https://repository.apache.org/content/repositories/orgapache
>> hadoop-1090/
>> >> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>> >>
>> >> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> >> include the first class GPU/FPGA support on YARN, Native services,
>> Support
>> >> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> >> block replicas to be provided by an external storage system, etc.
>> >>
>> >> For 3.1.0 RC0 vote discussion, please see [3].
>> >>
>> >> We’d like to use this as a starting release for 3.1.x [1], depending on
>> >> how
>> >> it goes, get it stabilized and potentially use a 3.1.1 in several
>> weeks as
>> >> the stable release.
>> >>
>> >> We have done testing with a pseudo cluster:
>> >> - Ran distributed job.
>> >> - GPU scheduling/isolation.
>> >> - Placement constraints (intra-application anti-affinity) by using
>> >> distributed shell.
>> >>
>> >> My +1 to start.
>> >>
>> >> Best,
>> >> Wangda/Vinod
>> >>
>> >> [1]
>> >> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> >> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> >> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>> (3.1.0)
>> >> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>> >> fixVersion ASC
>> >> [3]
>> >> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> >> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>> >>
>> >
>>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Thanks @Konstantin for reporting this issue, I will post comments on the
JIRA (HADOOP-15205)
- Wangda
On Tue, Apr 10, 2018 at 12:08 PM, Konstantin Shvachko <sh...@gmail.com>
wrote:
> A note to release managers. As discussed in
> https://issues.apache.org/jira/browse/HADOOP-15205
> We are producing release artifacts without sources jars. See e.g.
> https://repository.apache.org/content/repositories/releases/
> org/apache/hadoop/hadoop-common/3.1.0/
> I believe this has something to do with maven deployment stage,
> potentially maven-source-plugin.
> This is similar for all releases now, and I believe it should be fixed.
>
> Thanks,
> --Konstantin
>
> On Fri, Apr 6, 2018 at 2:01 PM, Wangda Tan <wh...@gmail.com> wrote:
>
>> Thanks guys for the additional votes! I just sent out announcement email.
>>
>> Best,
>> Wangda
>>
>> On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵 <ju...@apache.org> wrote:
>>
>> > Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
>> > based on:
>> >
>> > - Verified signatures
>> >
>> > - Verified checksums for source and binary artifacts
>> >
>> > - Built from source
>> >
>> > - Deployed a single node cluster
>> >
>> > - Verified web UIs, include Namenode, RM, etc.
>> >
>> > * Tried shell commands of HDFS and YARN
>> >
>> > * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
>> >
>> >
>> > Thanks,
>> >
>> >
>> > Junping
>> >
>> >
>> >
>> > Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
>> >
>> >> Hi folks,
>> >>
>> >> Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>> >> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>> >>
>> >> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>> >>
>> >> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> >> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>> >>
>> >> The maven artifacts are available via repository.apache.org at
>> >> https://repository.apache.org/content/repositories/orgapache
>> hadoop-1090/
>> >> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>> >>
>> >> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> >> include the first class GPU/FPGA support on YARN, Native services,
>> Support
>> >> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> >> block replicas to be provided by an external storage system, etc.
>> >>
>> >> For 3.1.0 RC0 vote discussion, please see [3].
>> >>
>> >> We’d like to use this as a starting release for 3.1.x [1], depending on
>> >> how
>> >> it goes, get it stabilized and potentially use a 3.1.1 in several
>> weeks as
>> >> the stable release.
>> >>
>> >> We have done testing with a pseudo cluster:
>> >> - Ran distributed job.
>> >> - GPU scheduling/isolation.
>> >> - Placement constraints (intra-application anti-affinity) by using
>> >> distributed shell.
>> >>
>> >> My +1 to start.
>> >>
>> >> Best,
>> >> Wangda/Vinod
>> >>
>> >> [1]
>> >> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> >> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> >> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>> (3.1.0)
>> >> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>> >> fixVersion ASC
>> >> [3]
>> >> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> >> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>> >>
>> >
>>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Thanks @Konstantin for reporting this issue, I will post comments on the
JIRA (HADOOP-15205)
- Wangda
On Tue, Apr 10, 2018 at 12:08 PM, Konstantin Shvachko <sh...@gmail.com>
wrote:
> A note to release managers. As discussed in
> https://issues.apache.org/jira/browse/HADOOP-15205
> We are producing release artifacts without sources jars. See e.g.
> https://repository.apache.org/content/repositories/releases/
> org/apache/hadoop/hadoop-common/3.1.0/
> I believe this has something to do with maven deployment stage,
> potentially maven-source-plugin.
> This is similar for all releases now, and I believe it should be fixed.
>
> Thanks,
> --Konstantin
>
> On Fri, Apr 6, 2018 at 2:01 PM, Wangda Tan <wh...@gmail.com> wrote:
>
>> Thanks guys for the additional votes! I just sent out announcement email.
>>
>> Best,
>> Wangda
>>
>> On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵 <ju...@apache.org> wrote:
>>
>> > Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
>> > based on:
>> >
>> > - Verified signatures
>> >
>> > - Verified checksums for source and binary artifacts
>> >
>> > - Built from source
>> >
>> > - Deployed a single node cluster
>> >
>> > - Verified web UIs, include Namenode, RM, etc.
>> >
>> > * Tried shell commands of HDFS and YARN
>> >
>> > * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
>> >
>> >
>> > Thanks,
>> >
>> >
>> > Junping
>> >
>> >
>> >
>> > Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
>> >
>> >> Hi folks,
>> >>
>> >> Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>> >> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>> >>
>> >> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>> >>
>> >> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> >> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>> >>
>> >> The maven artifacts are available via repository.apache.org at
>> >> https://repository.apache.org/content/repositories/orgapache
>> hadoop-1090/
>> >> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>> >>
>> >> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> >> include the first class GPU/FPGA support on YARN, Native services,
>> Support
>> >> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> >> block replicas to be provided by an external storage system, etc.
>> >>
>> >> For 3.1.0 RC0 vote discussion, please see [3].
>> >>
>> >> We’d like to use this as a starting release for 3.1.x [1], depending on
>> >> how
>> >> it goes, get it stabilized and potentially use a 3.1.1 in several
>> weeks as
>> >> the stable release.
>> >>
>> >> We have done testing with a pseudo cluster:
>> >> - Ran distributed job.
>> >> - GPU scheduling/isolation.
>> >> - Placement constraints (intra-application anti-affinity) by using
>> >> distributed shell.
>> >>
>> >> My +1 to start.
>> >>
>> >> Best,
>> >> Wangda/Vinod
>> >>
>> >> [1]
>> >> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> >> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> >> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>> (3.1.0)
>> >> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>> >> fixVersion ASC
>> >> [3]
>> >> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> >> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>> >>
>> >
>>
>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Konstantin Shvachko <sh...@gmail.com>.
A note to release managers. As discussed in
https://issues.apache.org/jira/browse/HADOOP-15205
We are producing release artifacts without sources jars. See e.g.
https://repository.apache.org/content/repositories/releases/org/apache/hadoop/hadoop-common/3.1.0/
I believe this has something to do with maven deployment stage, potentially
maven-source-plugin.
This is similar for all releases now, and I believe it should be fixed.
Thanks,
--Konstantin
On Fri, Apr 6, 2018 at 2:01 PM, Wangda Tan <wh...@gmail.com> wrote:
> Thanks guys for the additional votes! I just sent out announcement email.
>
> Best,
> Wangda
>
> On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵 <ju...@apache.org> wrote:
>
> > Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
> > based on:
> >
> > - Verified signatures
> >
> > - Verified checksums for source and binary artifacts
> >
> > - Built from source
> >
> > - Deployed a single node cluster
> >
> > - Verified web UIs, include Namenode, RM, etc.
> >
> > * Tried shell commands of HDFS and YARN
> >
> > * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
> >
> >
> > Thanks,
> >
> >
> > Junping
> >
> >
> >
> > Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
> >
> >> Hi folks,
> >>
> >> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
> >> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >>
> >> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >>
> >> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> >> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >>
> >> The maven artifacts are available via repository.apache.org at
> >> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
> >> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >>
> >> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> >> include the first class GPU/FPGA support on YARN, Native services,
> Support
> >> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> >> block replicas to be provided by an external storage system, etc.
> >>
> >> For 3.1.0 RC0 vote discussion, please see [3].
> >>
> >> We’d like to use this as a starting release for 3.1.x [1], depending on
> >> how
> >> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
> >> the stable release.
> >>
> >> We have done testing with a pseudo cluster:
> >> - Ran distributed job.
> >> - GPU scheduling/isolation.
> >> - Placement constraints (intra-application anti-affinity) by using
> >> distributed shell.
> >>
> >> My +1 to start.
> >>
> >> Best,
> >> Wangda/Vinod
> >>
> >> [1]
> >> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> >> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> >> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> >> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
> >> fixVersion ASC
> >> [3]
> >> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> >> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Konstantin Shvachko <sh...@gmail.com>.
A note to release managers. As discussed in
https://issues.apache.org/jira/browse/HADOOP-15205
We are producing release artifacts without sources jars. See e.g.
https://repository.apache.org/content/repositories/releases/org/apache/hadoop/hadoop-common/3.1.0/
I believe this has something to do with maven deployment stage, potentially
maven-source-plugin.
This is similar for all releases now, and I believe it should be fixed.
Thanks,
--Konstantin
On Fri, Apr 6, 2018 at 2:01 PM, Wangda Tan <wh...@gmail.com> wrote:
> Thanks guys for the additional votes! I just sent out announcement email.
>
> Best,
> Wangda
>
> On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵 <ju...@apache.org> wrote:
>
> > Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
> > based on:
> >
> > - Verified signatures
> >
> > - Verified checksums for source and binary artifacts
> >
> > - Built from source
> >
> > - Deployed a single node cluster
> >
> > - Verified web UIs, include Namenode, RM, etc.
> >
> > * Tried shell commands of HDFS and YARN
> >
> > * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
> >
> >
> > Thanks,
> >
> >
> > Junping
> >
> >
> >
> > Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
> >
> >> Hi folks,
> >>
> >> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
> >> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >>
> >> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >>
> >> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> >> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >>
> >> The maven artifacts are available via repository.apache.org at
> >> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
> >> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >>
> >> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> >> include the first class GPU/FPGA support on YARN, Native services,
> Support
> >> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> >> block replicas to be provided by an external storage system, etc.
> >>
> >> For 3.1.0 RC0 vote discussion, please see [3].
> >>
> >> We’d like to use this as a starting release for 3.1.x [1], depending on
> >> how
> >> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
> >> the stable release.
> >>
> >> We have done testing with a pseudo cluster:
> >> - Ran distributed job.
> >> - GPU scheduling/isolation.
> >> - Placement constraints (intra-application anti-affinity) by using
> >> distributed shell.
> >>
> >> My +1 to start.
> >>
> >> Best,
> >> Wangda/Vinod
> >>
> >> [1]
> >> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> >> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> >> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> >> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
> >> fixVersion ASC
> >> [3]
> >> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> >> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Konstantin Shvachko <sh...@gmail.com>.
A note to release managers. As discussed in
https://issues.apache.org/jira/browse/HADOOP-15205
We are producing release artifacts without sources jars. See e.g.
https://repository.apache.org/content/repositories/releases/org/apache/hadoop/hadoop-common/3.1.0/
I believe this has something to do with maven deployment stage, potentially
maven-source-plugin.
This is similar for all releases now, and I believe it should be fixed.
Thanks,
--Konstantin
On Fri, Apr 6, 2018 at 2:01 PM, Wangda Tan <wh...@gmail.com> wrote:
> Thanks guys for the additional votes! I just sent out announcement email.
>
> Best,
> Wangda
>
> On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵 <ju...@apache.org> wrote:
>
> > Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
> > based on:
> >
> > - Verified signatures
> >
> > - Verified checksums for source and binary artifacts
> >
> > - Built from source
> >
> > - Deployed a single node cluster
> >
> > - Verified web UIs, include Namenode, RM, etc.
> >
> > * Tried shell commands of HDFS and YARN
> >
> > * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
> >
> >
> > Thanks,
> >
> >
> > Junping
> >
> >
> >
> > Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
> >
> >> Hi folks,
> >>
> >> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
> >> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >>
> >> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >>
> >> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> >> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >>
> >> The maven artifacts are available via repository.apache.org at
> >> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
> >> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >>
> >> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> >> include the first class GPU/FPGA support on YARN, Native services,
> Support
> >> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> >> block replicas to be provided by an external storage system, etc.
> >>
> >> For 3.1.0 RC0 vote discussion, please see [3].
> >>
> >> We’d like to use this as a starting release for 3.1.x [1], depending on
> >> how
> >> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
> >> the stable release.
> >>
> >> We have done testing with a pseudo cluster:
> >> - Ran distributed job.
> >> - GPU scheduling/isolation.
> >> - Placement constraints (intra-application anti-affinity) by using
> >> distributed shell.
> >>
> >> My +1 to start.
> >>
> >> Best,
> >> Wangda/Vinod
> >>
> >> [1]
> >> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> >> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> >> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> >> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
> >> fixVersion ASC
> >> [3]
> >> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> >> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Konstantin Shvachko <sh...@gmail.com>.
A note to release managers. As discussed in
https://issues.apache.org/jira/browse/HADOOP-15205
We are producing release artifacts without sources jars. See e.g.
https://repository.apache.org/content/repositories/releases/org/apache/hadoop/hadoop-common/3.1.0/
I believe this has something to do with maven deployment stage, potentially
maven-source-plugin.
This is similar for all releases now, and I believe it should be fixed.
Thanks,
--Konstantin
On Fri, Apr 6, 2018 at 2:01 PM, Wangda Tan <wh...@gmail.com> wrote:
> Thanks guys for the additional votes! I just sent out announcement email.
>
> Best,
> Wangda
>
> On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵 <ju...@apache.org> wrote:
>
> > Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
> > based on:
> >
> > - Verified signatures
> >
> > - Verified checksums for source and binary artifacts
> >
> > - Built from source
> >
> > - Deployed a single node cluster
> >
> > - Verified web UIs, include Namenode, RM, etc.
> >
> > * Tried shell commands of HDFS and YARN
> >
> > * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
> >
> >
> > Thanks,
> >
> >
> > Junping
> >
> >
> >
> > Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
> >
> >> Hi folks,
> >>
> >> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
> >> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >>
> >> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >>
> >> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> >> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >>
> >> The maven artifacts are available via repository.apache.org at
> >> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
> >> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >>
> >> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> >> include the first class GPU/FPGA support on YARN, Native services,
> Support
> >> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> >> block replicas to be provided by an external storage system, etc.
> >>
> >> For 3.1.0 RC0 vote discussion, please see [3].
> >>
> >> We’d like to use this as a starting release for 3.1.x [1], depending on
> >> how
> >> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
> >> the stable release.
> >>
> >> We have done testing with a pseudo cluster:
> >> - Ran distributed job.
> >> - GPU scheduling/isolation.
> >> - Placement constraints (intra-application anti-affinity) by using
> >> distributed shell.
> >>
> >> My +1 to start.
> >>
> >> Best,
> >> Wangda/Vinod
> >>
> >> [1]
> >> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> >> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> >> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> >> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
> >> fixVersion ASC
> >> [3]
> >> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> >> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
> >>
> >
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Thanks guys for the additional votes! I just sent out announcement email.
Best,
Wangda
On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵 <ju...@apache.org> wrote:
> Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
> based on:
>
> - Verified signatures
>
> - Verified checksums for source and binary artifacts
>
> - Built from source
>
> - Deployed a single node cluster
>
> - Verified web UIs, include Namenode, RM, etc.
>
> * Tried shell commands of HDFS and YARN
>
> * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
>
>
> Thanks,
>
>
> Junping
>
>
>
> Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1]. We've
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services, Support
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
>> fixVersion ASC
>> [3]
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Thanks guys for the additional votes! I just sent out announcement email.
Best,
Wangda
On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵 <ju...@apache.org> wrote:
> Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
> based on:
>
> - Verified signatures
>
> - Verified checksums for source and binary artifacts
>
> - Built from source
>
> - Deployed a single node cluster
>
> - Verified web UIs, include Namenode, RM, etc.
>
> * Tried shell commands of HDFS and YARN
>
> * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
>
>
> Thanks,
>
>
> Junping
>
>
>
> Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1]. We've
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services, Support
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
>> fixVersion ASC
>> [3]
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Thanks guys for the additional votes! I just sent out announcement email.
Best,
Wangda
On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵 <ju...@apache.org> wrote:
> Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
> based on:
>
> - Verified signatures
>
> - Verified checksums for source and binary artifacts
>
> - Built from source
>
> - Deployed a single node cluster
>
> - Verified web UIs, include Namenode, RM, etc.
>
> * Tried shell commands of HDFS and YARN
>
> * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
>
>
> Thanks,
>
>
> Junping
>
>
>
> Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1]. We've
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services, Support
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
>> fixVersion ASC
>> [3]
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Wangda Tan <wh...@gmail.com>.
Thanks guys for the additional votes! I just sent out announcement email.
Best,
Wangda
On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵 <ju...@apache.org> wrote:
> Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
> based on:
>
> - Verified signatures
>
> - Verified checksums for source and binary artifacts
>
> - Built from source
>
> - Deployed a single node cluster
>
> - Verified web UIs, include Namenode, RM, etc.
>
> * Tried shell commands of HDFS and YARN
>
> * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
>
>
> Thanks,
>
>
> Junping
>
>
>
> Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1]. We've
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services, Support
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
>> fixVersion ASC
>> [3]
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by 俊平堵 <ju...@apache.org>.
Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
based on:
- Verified signatures
- Verified checksums for source and binary artifacts
- Built from source
- Deployed a single node cluster
- Verified web UIs, include Namenode, RM, etc.
* Tried shell commands of HDFS and YARN
* Ran sample MR jobs, include PI, Sleep, Terasort, etc.
Thanks,
Junping
Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Konstantinos Karanasos <kk...@gmail.com>.
+1 (binding)
Thanks for putting this together, Wangda.
I deployed a 7-node hadoop cluster and:
- ran various MR jobs
- ran the same jobs with a mix of opportunistic containers via centralized
scheduling
- ran the jobs with opportunistic containers and distributed scheduling
- ran some jobs with various placement constraints.
All worked as expected.
Thanks,
Konstantinos
On Thu, Mar 29, 2018 at 9:15 PM, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by 俊平堵 <ju...@apache.org>.
Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
based on:
- Verified signatures
- Verified checksums for source and binary artifacts
- Built from source
- Deployed a single node cluster
- Verified web UIs, include Namenode, RM, etc.
* Tried shell commands of HDFS and YARN
* Ran sample MR jobs, include PI, Sleep, Terasort, etc.
Thanks,
Junping
Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by 俊平堵 <ju...@apache.org>.
Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
based on:
- Verified signatures
- Verified checksums for source and binary artifacts
- Built from source
- Deployed a single node cluster
- Verified web UIs, include Namenode, RM, etc.
* Tried shell commands of HDFS and YARN
* Ran sample MR jobs, include PI, Sleep, Terasort, etc.
Thanks,
Junping
Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Konstantinos Karanasos <kk...@gmail.com>.
+1 (binding)
Thanks for putting this together, Wangda.
I deployed a 7-node hadoop cluster and:
- ran various MR jobs
- ran the same jobs with a mix of opportunistic containers via centralized
scheduling
- ran the jobs with opportunistic containers and distributed scheduling
- ran some jobs with various placement constraints.
All worked as expected.
Thanks,
Konstantinos
On Thu, Mar 29, 2018 at 9:15 PM, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by 俊平堵 <ju...@apache.org>.
Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
based on:
- Verified signatures
- Verified checksums for source and binary artifacts
- Built from source
- Deployed a single node cluster
- Verified web UIs, include Namenode, RM, etc.
* Tried shell commands of HDFS and YARN
* Ran sample MR jobs, include PI, Sleep, Terasort, etc.
Thanks,
Junping
Wangda Tan <wh...@gmail.com>于2018年3月30日 周五下午12:15写道：
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Eric Payne <er...@yahoo.com.INVALID>.
+1 (binding)
Thanks Wangda for doing the work to produce this release.
I did the following to test the release:
- Built from source
- Installed on 6-node pseudo cluster
- Interacted with RM CLI and GUI
- Tested streaming jobs
- Tested yarn distributed shell jobs
- Tested Max AM Resource Percent
- Tested simple inter-queue preemption
- Tested priority first intra-queue preemption
- Tested userlimit first intra-queue preemption
Thanks,Eric Payne
===========================================================
On Thursday, March 29, 2018, 11:15:51 PM CDT, Wangda Tan <wh...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Sunil G <su...@apache.org>.
Thanks Wangda for initiating the release.
I tested this RC built from source file.
- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM UI.
- Below feature sanity is done
- Application priority
- Application timeout
- Intra Queue preemption with priority based
- DS based affinity tests to verify placement constraints.
- Tested basic NodeLabel scenarios.
- Added couple of labels to few of nodes and behavior is coming
correct.
- Verified old UI and new YARN UI for labels.
- Submitted apps to labelled cluster and it works fine.
- Also performed few cli commands related to nodelabel.
- Test basic HA cases and seems correct.
- Tested new YARN UI . All pages are getting loaded correctly.
- Sunil
On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Eric Payne <er...@yahoo.com.INVALID>.
+1 (binding)
Thanks Wangda for doing the work to produce this release.
I did the following to test the release:
- Built from source
- Installed on 6-node pseudo cluster
- Interacted with RM CLI and GUI
- Tested streaming jobs
- Tested yarn distributed shell jobs
- Tested Max AM Resource Percent
- Tested simple inter-queue preemption
- Tested priority first intra-queue preemption
- Tested userlimit first intra-queue preemption
Thanks,Eric Payne
===========================================================
On Thursday, March 29, 2018, 11:15:51 PM CDT, Wangda Tan <wh...@gmail.com> wrote:
Hi folks,
Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
http://people.apache.org/~wangda/hadoop-3.1.0-RC1
The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.
For 3.1.0 RC0 vote discussion, please see [3].
We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.
We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.
My +1 to start.
Best,
Wangda/Vinod
[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Konstantinos Karanasos <kk...@gmail.com>.
+1 (binding)
Thanks for putting this together, Wangda.
I deployed a 7-node hadoop cluster and:
- ran various MR jobs
- ran the same jobs with a mix of opportunistic containers via centralized
scheduling
- ran the jobs with opportunistic containers and distributed scheduling
- ran some jobs with various placement constraints.
All worked as expected.
Thanks,
Konstantinos
On Thu, Mar 29, 2018 at 9:15 PM, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Rohith Sharma K S <ro...@apache.org>.
+1 (binding)
* Downloaded source, built from source with -Dhbase.profile=2.0.
* Installed RM HA cluster integrated with ATSv2. Installed HBase-2.0-beta1
for ATSv2 back end. Scheduler is configured with 2 level queue hierarchy.
* Ran sample jobs such as MR/Distributed shell and verified for
** ATSv2 Timeline Reader REST API's. Validated for data published in ATSv2.
** YARN UI2 accessed for Flow Activity pages. Navigated inside flow
activity page for all other info.
** YARN old UI also verified for all the pages
** RM HA/Restart/work-preserving-restart are tested while running a job.
** NM restart scenarios are verified.
** Application timeout and application priority feature is tested.
Thanks & Regards
Rohith Sharma K S
On 30 March 2018 at 09:45, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Yufei Gu <fl...@gmail.com>.
Thanks Wangda for working on this!
+1 (non-binding)
- Downloaded the binary tarball and verified the checksum.
- Started a pseudo cluster inside one docker container
- Run Resource Manager with Fair Scheduler
- Verified distributed shell
- Verified mapreduce pi job
- Sanity checked RM WebUI
Best,
Yufei
On Thu, Mar 29, 2018 at 9:15 PM, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Posted by Brahma Reddy Battula <br...@gmail.com>.
Wangda thanks for driving this.
+1(binding)
--Built from source
--Installed HA cluster
--Verified Basic Shell commands
--Ran Sample Jobs
--Browsed the UI's.
On Fri, Mar 30, 2018 at 9:45 AM, Wangda Tan <wh...@gmail.com> wrote:
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>