You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Vinod Kumar Vavilapalli <vi...@apache.org> on 2016/08/12 16:45:28 UTC
[VOTE] Release Apache Hadoop 2.7.3 RC1
Hi all,
I've created a release candidate RC1 for Apache Hadoop 2.7.3.
As discussed before, this is the next maintenance release to follow up 2.7.2.
The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
The RC tag in git is: release-2.7.3-RC1
The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
As you may have noted,
- few issues with RC0 forced a RC1 [1]
- a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
Please try the release and vote; the vote will run for the usual 5 days.
Thanks,
Vinod
[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
[2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks Steve, this is one area that isn’t very well release-tested usually!
+Vinod
> On Aug 16, 2016, at 2:25 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> I've just looked at the staged JARs and how they worked with downstream apps —that being a key way that Hadoop artifacts are adopted.
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks Steve, this is one area that isn’t very well release-tested usually!
+Vinod
> On Aug 16, 2016, at 2:25 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> I've just looked at the staged JARs and how they worked with downstream apps —that being a key way that Hadoop artifacts are adopted.
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks Steve, this is one area that isn’t very well release-tested usually!
+Vinod
> On Aug 16, 2016, at 2:25 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> I've just looked at the staged JARs and how they worked with downstream apps —that being a key way that Hadoop artifacts are adopted.
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks Steve, this is one area that isn’t very well release-tested usually!
+Vinod
> On Aug 16, 2016, at 2:25 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> I've just looked at the staged JARs and how they worked with downstream apps —that being a key way that Hadoop artifacts are adopted.
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Steve Loughran <st...@hortonworks.com>.
+1 binding
1. built and tested apache slider (incubating) against the Hadoop 2.7.3 artifacts
2. did a build & test of Apache Spark master branch iwth 2.7.3 JARs,
For that I had to tweak spark's build to support the staging repo; hopefully that will get into Spark
https://issues.apache.org/jira/browse/SPARK-17058
3. did a test run of my WiP SPARK-7481 spark-cloud module; after fixing a couple of things on the test setup side related to HADOOP-13058,
mvn test --pl cloud -Pyarn,hadoop-2.7,snapshots-and-staging -Dhadoop.version=2.7.3 -Dcloud.test.configuration.file=../conf/cloud-tests.xml
all was well —albeit measurably slower than Hadoop 2.8. That's proof that the 2.8 version of s3a really does deliver measurable speedup for those tests (currently just file input/seek; more to come). I had originally thought things were broken as s3 init was failing -but that's because the s3 bucket was in frankfurt, and the AWS library used can't talk to that endpoint (v4 auth protocol, see).
4. did a full spark distribution build of that SPARK-7481 branch
dev/make-distribution.sh -Pyarn,hadoop-2.7,snapshots-and-staging -Dhadoop.version=2.7.3
ran command line test to do read of s3a data:
bin/spark-submit --class org.apache.spark.cloud.s3.examples.S3LineCount \
--conf spark.hadoop.fs.s3a.access.key=$AWS_KEY \
--conf spark.hadoop.fs.s3a.secret.key=$AWS_SECRET \
examples/jars/spark-examples_2.11-2.1.0-SNAPSHOT.jar
5. Pulled out the microsoft Azure JAR azure-storage-2.0.0.jar and repeated step 4
-this showed that the 2.7.x branch does handle the failure to load a filesystem due to dependency or other classloading problems —this was proving a big problem in adding the aws & azure stuff to the spark build, as it'd stop spark from starting up if the dependencies were absent.
I've not done any of the .tar.gz diligence; I've just looked at the staged JARs and how they worked with downstream apps —that being a key way that Hadoop artifacts are adopted.
> On 12 Aug 2016, at 17:45, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Jason Lowe <jl...@yahoo-inc.com.INVALID>.
+1 (binding)
- Verified signatures and digests- Built from source with native support- Deployed a pseudo-distributed cluster- Ran some sample jobs
Jason
From: Vinod Kumar Vavilapalli <vi...@apache.org>
To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
Cc: Vinod Kumar Vavilapalli <vi...@apache.org>
Sent: Friday, August 12, 2016 11:45 AM
Subject: [VOTE] Release Apache Hadoop 2.7.3 RC1
Hi all,
I've created a release candidate RC1 for Apache Hadoop 2.7.3.
As discussed before, this is the next maintenance release to follow up 2.7.2.
The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
The RC tag in git is: release-2.7.3-RC1
The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
As you may have noted,
- few issues with RC0 forced a RC1 [1]
- a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
Please try the release and vote; the vote will run for the usual 5 days.
Thanks,
Vinod
[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
[2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks Marco. It was a Thursday late-night slip-up.
Fixed the dates and replaced the bits, so the voting can continue.
FYI, they aren’t binding though - as it all depends on how the release voting goes. One should usually only trust the release-date published on the website.
Thanks
+Vinod
> On Aug 13, 2016, at 1:35 PM, Marco Zühlke <mzuehlke@gmail.com <ma...@gmail.com>> wrote:
>
> Hi Vinod,
>
> I'm not sure if this is relevant, but you changed the release date in the CHANGES.txt <https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f#diff-4de1a6452466a82b89570bd9ab606c12> files to 2016-09-19.
> I guess you have meant 2016-08-19.
>
> See: https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f <https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f>
>
>
> Thanks,
> Marco
>
>
>
> 2016-08-12 18:45 GMT+02:00 Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>>:
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> <http://repository.apache.org/ <http://repository.apache.org/>> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html> <http://markmail.org/thread/6yv2fyrs4jlepmmr <http://markmail.org/thread/6yv2fyrs4jlepmmr>>
>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks Marco. It was a Thursday late-night slip-up.
Fixed the dates and replaced the bits, so the voting can continue.
FYI, they aren’t binding though - as it all depends on how the release voting goes. One should usually only trust the release-date published on the website.
Thanks
+Vinod
> On Aug 13, 2016, at 1:35 PM, Marco Zühlke <mzuehlke@gmail.com <ma...@gmail.com>> wrote:
>
> Hi Vinod,
>
> I'm not sure if this is relevant, but you changed the release date in the CHANGES.txt <https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f#diff-4de1a6452466a82b89570bd9ab606c12> files to 2016-09-19.
> I guess you have meant 2016-08-19.
>
> See: https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f <https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f>
>
>
> Thanks,
> Marco
>
>
>
> 2016-08-12 18:45 GMT+02:00 Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>>:
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> <http://repository.apache.org/ <http://repository.apache.org/>> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html> <http://markmail.org/thread/6yv2fyrs4jlepmmr <http://markmail.org/thread/6yv2fyrs4jlepmmr>>
>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks Marco. It was a Thursday late-night slip-up.
Fixed the dates and replaced the bits, so the voting can continue.
FYI, they aren’t binding though - as it all depends on how the release voting goes. One should usually only trust the release-date published on the website.
Thanks
+Vinod
> On Aug 13, 2016, at 1:35 PM, Marco Zühlke <mzuehlke@gmail.com <ma...@gmail.com>> wrote:
>
> Hi Vinod,
>
> I'm not sure if this is relevant, but you changed the release date in the CHANGES.txt <https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f#diff-4de1a6452466a82b89570bd9ab606c12> files to 2016-09-19.
> I guess you have meant 2016-08-19.
>
> See: https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f <https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f>
>
>
> Thanks,
> Marco
>
>
>
> 2016-08-12 18:45 GMT+02:00 Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>>:
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> <http://repository.apache.org/ <http://repository.apache.org/>> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html> <http://markmail.org/thread/6yv2fyrs4jlepmmr <http://markmail.org/thread/6yv2fyrs4jlepmmr>>
>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks Marco. It was a Thursday late-night slip-up.
Fixed the dates and replaced the bits, so the voting can continue.
FYI, they aren’t binding though - as it all depends on how the release voting goes. One should usually only trust the release-date published on the website.
Thanks
+Vinod
> On Aug 13, 2016, at 1:35 PM, Marco Zühlke <mzuehlke@gmail.com <ma...@gmail.com>> wrote:
>
> Hi Vinod,
>
> I'm not sure if this is relevant, but you changed the release date in the CHANGES.txt <https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f#diff-4de1a6452466a82b89570bd9ab606c12> files to 2016-09-19.
> I guess you have meant 2016-08-19.
>
> See: https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f <https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f>
>
>
> Thanks,
> Marco
>
>
>
> 2016-08-12 18:45 GMT+02:00 Vinod Kumar Vavilapalli <vinodkv@apache.org <ma...@apache.org>>:
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> <http://repository.apache.org/ <http://repository.apache.org/>> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html> <http://markmail.org/thread/6yv2fyrs4jlepmmr <http://markmail.org/thread/6yv2fyrs4jlepmmr>>
>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Marco Zühlke <mz...@gmail.com>.
Hi Vinod,
I'm not sure if this is relevant, but you changed the release date in the
CHANGES.txt
<https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f#diff-4de1a6452466a82b89570bd9ab606c12>
files to 2016-09-19.
I guess you have meant 2016-08-19.
See:
https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f
Thanks,
Marco
2016-08-12 18:45 GMT+02:00 Vinod Kumar Vavilapalli <vi...@apache.org>:
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up
> 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at https://repository.apache.org/
> content/repositories/orgapachehadoop-1045/ <https://repository.apache.
> org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893)
> caused 2.7.3 (along with every other Hadoop release) to slip by quite a
> bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/
> hdfs-dev%40hadoop.apache.org/index.html#26106 <
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106
> >
> [2]: 2.7.3 release plan: https://www.mail-archive.com/
> hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/
> 6yv2fyrs4jlepmmr>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Jason Lowe <jl...@yahoo-inc.com.INVALID>.
+1 (binding)
- Verified signatures and digests- Built from source with native support- Deployed a pseudo-distributed cluster- Ran some sample jobs
Jason
From: Vinod Kumar Vavilapalli <vi...@apache.org>
To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
Cc: Vinod Kumar Vavilapalli <vi...@apache.org>
Sent: Friday, August 12, 2016 11:45 AM
Subject: [VOTE] Release Apache Hadoop 2.7.3 RC1
Hi all,
I've created a release candidate RC1 for Apache Hadoop 2.7.3.
As discussed before, this is the next maintenance release to follow up 2.7.2.
The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
The RC tag in git is: release-2.7.3-RC1
The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
As you may have noted,
- few issues with RC0 forced a RC1 [1]
- a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
Please try the release and vote; the vote will run for the usual 5 days.
Thanks,
Vinod
[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
[2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
RE: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Balkrushna Patil <ba...@augmentiq.co.in>.
Hi all,
I want to go dipper in mapreduce source code so I have to embed that code in
IDE. Can anybody tell me from where I should have to download source code.
Thanks and Regards
Balkrushna Patil
augmentIQ Data Sciences Pvt Ltd.
Mob: +91-9766 4996 81
-----Original Message-----
From: Junping Du [mailto:jdu@hortonworks.com]
Sent: 17 August 2016 18:45
To: Allen Wittenauer; common-dev@hadoop.apache.org; kshukla@yahoo-inc.com;
kihwal@yahoo-inc.com
Cc: hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
From my quick understanding, HDFS-9395 is more like a bug fix and
improvement for audit logging instead of incompatible changes. We mark
incompatible probably because the audit log behavior could be
corrected/updated in some exception cases. I think it still belongs to 2.7.3
scope.
Kuhu and Kihwal, any comments here?
Thanks,
Junping
________________________________________
From: Allen Wittenauer <aw...@effectivemachines.com>
Sent: Wednesday, August 17, 2016 5:29 AM
To: common-dev@hadoop.apache.org
Cc: hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
-1
HDFS-9395 is an incompatible change:
a) Why is not marked as such in the changes file?
b) Why is an incompatible change in a micro release, much less a minor?
c) Where is the release note for this change?
> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org>
wrote:
>
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up
2.7.2.
>
> The RC is available for validation at:
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/
<http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org
<http://repository.apache.org/> at
https://repository.apache.org/content/repositories/orgapachehadoop-1045/
<https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html
<http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for
your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893)
caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit.
This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0:
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106
<https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
> [2]: 2.7.3 release plan:
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html
<http://markmail.org/thread/6yv2fyrs4jlepmmr>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
Touching the audit log is *extremely* dangerous from a compatibility perspective. It is easily the most machine processed log in Hadoop (with the second likely being the fsck log). In particular, this comment tells me that we are almost certainly going to break users:
"Some audit logs ( for non-ACE failures ) will go missing. So this change needs to be marked as Incompatible, for heads-up."
If that means what I think it means (the ordering of checks is going to make previously logged errors disappear in lieu of other, new messages showing up first), that is going to cause massive problems for users who are looking for a particular entry. Worse, while the JIRA was marked incompatible, there are absolutely zero hints to end users (changes file, release notes) that this could potentially break their universe without digging into the comments of said JIRA. That's not a heads up, that's a landmine.
It's also arguable that this is actually a bug fix. A lot of the assumptions made in that JIRA about the audit logs original intent are completely wrong. Better yet, a lot of the justification is around another unmarked, incompatible change that was introduced in the 2.x timeline.
Even if one disagrees and still views this as a bug fix: it's still an incompatible change. Users are justifiably angry when we don't warn them about breakages and this is a great example of that.
> On Aug 17, 2016, at 6:15 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> From my quick understanding, HDFS-9395 is more like a bug fix and improvement for audit logging instead of incompatible changes. We mark incompatible probably because the audit log behavior could be corrected/updated in some exception cases. I think it still belongs to 2.7.3 scope.
> Kuhu and Kihwal, any comments here?
>
>
> Thanks,
>
> Junping
> ________________________________________
> From: Allen Wittenauer <aw...@effectivemachines.com>
> Sent: Wednesday, August 17, 2016 5:29 AM
> To: common-dev@hadoop.apache.org
> Cc: hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
Touching the audit log is *extremely* dangerous from a compatibility perspective. It is easily the most machine processed log in Hadoop (with the second likely being the fsck log). In particular, this comment tells me that we are almost certainly going to break users:
"Some audit logs ( for non-ACE failures ) will go missing. So this change needs to be marked as Incompatible, for heads-up."
If that means what I think it means (the ordering of checks is going to make previously logged errors disappear in lieu of other, new messages showing up first), that is going to cause massive problems for users who are looking for a particular entry. Worse, while the JIRA was marked incompatible, there are absolutely zero hints to end users (changes file, release notes) that this could potentially break their universe without digging into the comments of said JIRA. That's not a heads up, that's a landmine.
It's also arguable that this is actually a bug fix. A lot of the assumptions made in that JIRA about the audit logs original intent are completely wrong. Better yet, a lot of the justification is around another unmarked, incompatible change that was introduced in the 2.x timeline.
Even if one disagrees and still views this as a bug fix: it's still an incompatible change. Users are justifiably angry when we don't warn them about breakages and this is a great example of that.
> On Aug 17, 2016, at 6:15 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> From my quick understanding, HDFS-9395 is more like a bug fix and improvement for audit logging instead of incompatible changes. We mark incompatible probably because the audit log behavior could be corrected/updated in some exception cases. I think it still belongs to 2.7.3 scope.
> Kuhu and Kihwal, any comments here?
>
>
> Thanks,
>
> Junping
> ________________________________________
> From: Allen Wittenauer <aw...@effectivemachines.com>
> Sent: Wednesday, August 17, 2016 5:29 AM
> To: common-dev@hadoop.apache.org
> Cc: hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
Touching the audit log is *extremely* dangerous from a compatibility perspective. It is easily the most machine processed log in Hadoop (with the second likely being the fsck log). In particular, this comment tells me that we are almost certainly going to break users:
"Some audit logs ( for non-ACE failures ) will go missing. So this change needs to be marked as Incompatible, for heads-up."
If that means what I think it means (the ordering of checks is going to make previously logged errors disappear in lieu of other, new messages showing up first), that is going to cause massive problems for users who are looking for a particular entry. Worse, while the JIRA was marked incompatible, there are absolutely zero hints to end users (changes file, release notes) that this could potentially break their universe without digging into the comments of said JIRA. That's not a heads up, that's a landmine.
It's also arguable that this is actually a bug fix. A lot of the assumptions made in that JIRA about the audit logs original intent are completely wrong. Better yet, a lot of the justification is around another unmarked, incompatible change that was introduced in the 2.x timeline.
Even if one disagrees and still views this as a bug fix: it's still an incompatible change. Users are justifiably angry when we don't warn them about breakages and this is a great example of that.
> On Aug 17, 2016, at 6:15 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> From my quick understanding, HDFS-9395 is more like a bug fix and improvement for audit logging instead of incompatible changes. We mark incompatible probably because the audit log behavior could be corrected/updated in some exception cases. I think it still belongs to 2.7.3 scope.
> Kuhu and Kihwal, any comments here?
>
>
> Thanks,
>
> Junping
> ________________________________________
> From: Allen Wittenauer <aw...@effectivemachines.com>
> Sent: Wednesday, August 17, 2016 5:29 AM
> To: common-dev@hadoop.apache.org
> Cc: hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
Touching the audit log is *extremely* dangerous from a compatibility perspective. It is easily the most machine processed log in Hadoop (with the second likely being the fsck log). In particular, this comment tells me that we are almost certainly going to break users:
"Some audit logs ( for non-ACE failures ) will go missing. So this change needs to be marked as Incompatible, for heads-up."
If that means what I think it means (the ordering of checks is going to make previously logged errors disappear in lieu of other, new messages showing up first), that is going to cause massive problems for users who are looking for a particular entry. Worse, while the JIRA was marked incompatible, there are absolutely zero hints to end users (changes file, release notes) that this could potentially break their universe without digging into the comments of said JIRA. That's not a heads up, that's a landmine.
It's also arguable that this is actually a bug fix. A lot of the assumptions made in that JIRA about the audit logs original intent are completely wrong. Better yet, a lot of the justification is around another unmarked, incompatible change that was introduced in the 2.x timeline.
Even if one disagrees and still views this as a bug fix: it's still an incompatible change. Users are justifiably angry when we don't warn them about breakages and this is a great example of that.
> On Aug 17, 2016, at 6:15 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> From my quick understanding, HDFS-9395 is more like a bug fix and improvement for audit logging instead of incompatible changes. We mark incompatible probably because the audit log behavior could be corrected/updated in some exception cases. I think it still belongs to 2.7.3 scope.
> Kuhu and Kihwal, any comments here?
>
>
> Thanks,
>
> Junping
> ________________________________________
> From: Allen Wittenauer <aw...@effectivemachines.com>
> Sent: Wednesday, August 17, 2016 5:29 AM
> To: common-dev@hadoop.apache.org
> Cc: hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Junping Du <jd...@hortonworks.com>.
From my quick understanding, HDFS-9395 is more like a bug fix and improvement for audit logging instead of incompatible changes. We mark incompatible probably because the audit log behavior could be corrected/updated in some exception cases. I think it still belongs to 2.7.3 scope.
Kuhu and Kihwal, any comments here?
Thanks,
Junping
________________________________________
From: Allen Wittenauer <aw...@effectivemachines.com>
Sent: Wednesday, August 17, 2016 5:29 AM
To: common-dev@hadoop.apache.org
Cc: hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
-1
HDFS-9395 is an incompatible change:
a) Why is not marked as such in the changes file?
b) Why is an incompatible change in a micro release, much less a minor?
c) Where is the release note for this change?
> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by larry mccay <la...@gmail.com>.
I believe it was described as some previous audit entries have been
superseded by new ones and that the order may no longer be the same for
other entries.
For what it’s worth, I agree with the assertion that this is a backward
incompatible output - especially for audit logs.
On Thu, Aug 18, 2016 at 11:32 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 18 Aug 2016, at 14:57, Junping Du <jd...@hortonworks.com> wrote:
> >
> > I think Allen's previous comments are very misleading.
> > In my understanding, only incompatible API (RPC, CLIs, WebService, etc.)
> shouldn't land on branch-2, but other incompatible behaviors (logs,
> audit-log, daemon's restart, etc.) should get flexible for landing.
> Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with
> incompatible-changes could get landed on branch-2 after 2.2.0 release? Most
> of them are already released.
> >
> > Thanks,
> >
> > Junping
>
>
> Don't get AW started on compatiblity; it'll only upset him.
>
> One thing he does care about is the ability of programs to consume the
> output of commands and logs —and for that even the output of commands and
> logs need to continue to be parseable
>
> https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/
> Compatibility.html#Command_Line_Interface_CLI
>
> " Changing the path of a command, removing or renaming command line
> options, the order of arguments, or the command return code and output
> break compatibility and may adversely affect users."
>
> I believe Allen is particularly concerned that a minor point release is
> going in as incompatible, on the basis the audit log output will change
> —that's the log that is explicitly designed for machine processing, hooking
> up to flume & kafka, etc. As example, Spotify spoke at a Hadoop Summit
> conference about how they used it to identify files which hadn't been used
> for a long time; inferring an atime attribute from the access history.
>
> What has changed in the output?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by larry mccay <la...@gmail.com>.
I believe it was described as some previous audit entries have been
superseded by new ones and that the order may no longer be the same for
other entries.
For what it’s worth, I agree with the assertion that this is a backward
incompatible output - especially for audit logs.
On Thu, Aug 18, 2016 at 11:32 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 18 Aug 2016, at 14:57, Junping Du <jd...@hortonworks.com> wrote:
> >
> > I think Allen's previous comments are very misleading.
> > In my understanding, only incompatible API (RPC, CLIs, WebService, etc.)
> shouldn't land on branch-2, but other incompatible behaviors (logs,
> audit-log, daemon's restart, etc.) should get flexible for landing.
> Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with
> incompatible-changes could get landed on branch-2 after 2.2.0 release? Most
> of them are already released.
> >
> > Thanks,
> >
> > Junping
>
>
> Don't get AW started on compatiblity; it'll only upset him.
>
> One thing he does care about is the ability of programs to consume the
> output of commands and logs —and for that even the output of commands and
> logs need to continue to be parseable
>
> https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/
> Compatibility.html#Command_Line_Interface_CLI
>
> " Changing the path of a command, removing or renaming command line
> options, the order of arguments, or the command return code and output
> break compatibility and may adversely affect users."
>
> I believe Allen is particularly concerned that a minor point release is
> going in as incompatible, on the basis the audit log output will change
> —that's the log that is explicitly designed for machine processing, hooking
> up to flume & kafka, etc. As example, Spotify spoke at a Hadoop Summit
> conference about how they used it to identify files which hadn't been used
> for a long time; inferring an atime attribute from the access history.
>
> What has changed in the output?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by larry mccay <la...@gmail.com>.
I believe it was described as some previous audit entries have been
superseded by new ones and that the order may no longer be the same for
other entries.
For what it’s worth, I agree with the assertion that this is a backward
incompatible output - especially for audit logs.
On Thu, Aug 18, 2016 at 11:32 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 18 Aug 2016, at 14:57, Junping Du <jd...@hortonworks.com> wrote:
> >
> > I think Allen's previous comments are very misleading.
> > In my understanding, only incompatible API (RPC, CLIs, WebService, etc.)
> shouldn't land on branch-2, but other incompatible behaviors (logs,
> audit-log, daemon's restart, etc.) should get flexible for landing.
> Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with
> incompatible-changes could get landed on branch-2 after 2.2.0 release? Most
> of them are already released.
> >
> > Thanks,
> >
> > Junping
>
>
> Don't get AW started on compatiblity; it'll only upset him.
>
> One thing he does care about is the ability of programs to consume the
> output of commands and logs —and for that even the output of commands and
> logs need to continue to be parseable
>
> https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/
> Compatibility.html#Command_Line_Interface_CLI
>
> " Changing the path of a command, removing or renaming command line
> options, the order of arguments, or the command return code and output
> break compatibility and may adversely affect users."
>
> I believe Allen is particularly concerned that a minor point release is
> going in as incompatible, on the basis the audit log output will change
> —that's the log that is explicitly designed for machine processing, hooking
> up to flume & kafka, etc. As example, Spotify spoke at a Hadoop Summit
> conference about how they used it to identify files which hadn't been used
> for a long time; inferring an atime attribute from the access history.
>
> What has changed in the output?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by larry mccay <la...@gmail.com>.
I believe it was described as some previous audit entries have been
superseded by new ones and that the order may no longer be the same for
other entries.
For what it’s worth, I agree with the assertion that this is a backward
incompatible output - especially for audit logs.
On Thu, Aug 18, 2016 at 11:32 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 18 Aug 2016, at 14:57, Junping Du <jd...@hortonworks.com> wrote:
> >
> > I think Allen's previous comments are very misleading.
> > In my understanding, only incompatible API (RPC, CLIs, WebService, etc.)
> shouldn't land on branch-2, but other incompatible behaviors (logs,
> audit-log, daemon's restart, etc.) should get flexible for landing.
> Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with
> incompatible-changes could get landed on branch-2 after 2.2.0 release? Most
> of them are already released.
> >
> > Thanks,
> >
> > Junping
>
>
> Don't get AW started on compatiblity; it'll only upset him.
>
> One thing he does care about is the ability of programs to consume the
> output of commands and logs —and for that even the output of commands and
> logs need to continue to be parseable
>
> https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/
> Compatibility.html#Command_Line_Interface_CLI
>
> " Changing the path of a command, removing or renaming command line
> options, the order of arguments, or the command return code and output
> break compatibility and may adversely affect users."
>
> I believe Allen is particularly concerned that a minor point release is
> going in as incompatible, on the basis the audit log output will change
> —that's the log that is explicitly designed for machine processing, hooking
> up to flume & kafka, etc. As example, Spotify spoke at a Hadoop Summit
> conference about how they used it to identify files which hadn't been used
> for a long time; inferring an atime attribute from the access history.
>
> What has changed in the output?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Steve Loughran <st...@hortonworks.com>.
> On 18 Aug 2016, at 14:57, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
Don't get AW started on compatiblity; it'll only upset him.
One thing he does care about is the ability of programs to consume the output of commands and logs —and for that even the output of commands and logs need to continue to be parseable
https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/Compatibility.html#Command_Line_Interface_CLI
" Changing the path of a command, removing or renaming command line options, the order of arguments, or the command return code and output break compatibility and may adversely affect users."
I believe Allen is particularly concerned that a minor point release is going in as incompatible, on the basis the audit log output will change —that's the log that is explicitly designed for machine processing, hooking up to flume & kafka, etc. As example, Spotify spoke at a Hadoop Summit conference about how they used it to identify files which hadn't been used for a long time; inferring an atime attribute from the access history.
What has changed in the output?
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Steve Loughran <st...@hortonworks.com>.
> On 18 Aug 2016, at 14:57, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
Don't get AW started on compatiblity; it'll only upset him.
One thing he does care about is the ability of programs to consume the output of commands and logs —and for that even the output of commands and logs need to continue to be parseable
https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/Compatibility.html#Command_Line_Interface_CLI
" Changing the path of a command, removing or renaming command line options, the order of arguments, or the command return code and output break compatibility and may adversely affect users."
I believe Allen is particularly concerned that a minor point release is going in as incompatible, on the basis the audit log output will change —that's the log that is explicitly designed for machine processing, hooking up to flume & kafka, etc. As example, Spotify spoke at a Hadoop Summit conference about how they used it to identify files which hadn't been used for a long time; inferring an atime attribute from the access history.
What has changed in the output?
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Andrew Purtell <an...@gmail.com>.
>
What is a realistic strategy for us to evolve the HDFS audit log in a
backward-compatible way? If the API is essentially any form of ad-hoc
scripting, then for any proposed audit log format change, I can find a
reason to veto it on grounds of backward incompatibility.
Yeah when log scraping is the only way at information, then the API surface
expands to cover all manner of ad-hoc scripting.
Not sure moving away from emitting audit information in log lines would be
operator friendly. That's a tough one. Just about everything in the
ecosystem emits audit information as log lines. If Hadoop switches strategy
to become a one-off doing something different this would be painful.
Assuming log lines will be the way we continue to receive audit events from
Hadoop/HDFS, please consider freezing any changes to audit logging today,
develop a formal specification, add the specification to documentation, and
then take care to not break the specification between releases. Because
audit logging from the NN comes from low level places in FSNameSystem this
is going to constrain maintenance and refactor of that and related code, so
with my software maintainer hat on I feel your pain in advance. You'll want
to hash out what level of compatibility you'd like to offer. I'd recommend
only changing on major releases.
On Thu, Aug 18, 2016 at 10:04 AM, Chris Nauroth <cn...@hortonworks.com>
wrote:
> Andrew, thanks for adding your perspective on this.
>
>
> What is a realistic strategy for us to evolve the HDFS audit log in a
> backward-compatible way? If the API is essentially any form of ad-hoc
> scripting, then for any proposed audit log format change, I can find a
> reason to veto it on grounds of backward incompatibility.
>
> - I can’t add a new field on the end, because that would break an awk
> script that uses $NF expecting to find a specific field.
> - I can’t prepend a new field, because that would break a "cut -f1"
> expecting to find the timestamp.
> - HDFS can’t add any new features, because someone might have written a
> script that does "exit 1" if it finds an unexpected RPC in the "cmd=" field.
> - Hadoop is not allowed to add full IPv6 support, because someone might
> have written a script that looks at the "ip=" field and parses it by IPv4
> syntax.
>
> On the CLI, a potential solution for evolving the output is to preserve
> the old format by default and only enable the new format if the user
> explicitly passes a new argument. What should we do for the audit log?
> Configuration flags in hdfs-site.xml? (That of course adds its own brand
> of complexity.)
>
>
> I’m particularly interested to hear potential solutions from people like
> Andrew and Allen who have been most vocal about the need for a stable
> format. Without a solution, this unfortunately devolves into the format
> being frozen within a major release line.
>
> We could benefit from getting a patch on the compatibility doc that
> addresses the HDFS audit log specifically.
>
> --Chris Nauroth
>
> On 8/18/16, 8:47 AM, "Andrew Purtell" <an...@gmail.com> wrote:
>
> An incompatible APIs change is developer unfriendly. An incompatible
> behavioral change is operator unfriendly. Historically, one dimension of
> incompatibility has had a lot more mindshare than the other. It's great
> that this might be changing for the better.
>
> Where I work when we move from one Hadoop 2.x minor to another we
> always spend time updating our deployment plans, alerting, log scraping,
> and related things due to changes. Some are debatable as if qualifying for
> the 'incompatible' designation. I think the audit logging change that
> triggered this discussion is a good example of one that does. If you want
> to audit HDFS actions those log emissions are your API. (Inotify doesn't
> offer access control events.) One has to code regular expressions for
> parsing them and reverse engineer under what circumstances an audit line is
> emitted so you can make assumptions about what transpired. Change either
> and you might break someone's automation for meeting industry or legal
> compliance obligations. Not a trivial matter. If you don't operate Hadoop
> in production you might not realize the implications of such a change. Glad
> to see Hadoop has community diversity to recognize it in some cases.
>
> > On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
> >
> > I think Allen's previous comments are very misleading.
> > In my understanding, only incompatible API (RPC, CLIs, WebService,
> etc.) shouldn't land on branch-2, but other incompatible behaviors (logs,
> audit-log, daemon's restart, etc.) should get flexible for landing.
> Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with
> incompatible-changes could get landed on branch-2 after 2.2.0 release? Most
> of them are already released.
> >
> > Thanks,
> >
> > Junping
> > ________________________________________
> > From: Vinod Kumar Vavilapalli <vi...@apache.org>
> > Sent: Wednesday, August 17, 2016 9:29 PM
> > To: Allen Wittenauer
> > Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
> >
> > I always look at CHANGES.txt entries for incompatible-changes and
> this JIRA obviously wasn’t there.
> >
> > Anyways, this shouldn’t be in any of branch-2.* as committers there
> clearly mentioned that this is an incompatible change.
> >
> > I am reverting the patch from branch-2* .
> >
> > Thanks
> > +Vinod
> >
> >> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <
> aw@effectivemachines.com> wrote:
> >>
> >>
> >>
> >> -1
> >>
> >> HDFS-9395 is an incompatible change:
> >>
> >> a) Why is not marked as such in the changes file?
> >> b) Why is an incompatible change in a micro release, much less a
> minor?
> >> c) Where is the release note for this change?
> >>
> >>
> >>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
> >>>
> >>> As discussed before, this is the next maintenance release to
> follow up 2.7.2.
> >>>
> >>> The RC is available for validation at: http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC0/>
> >>>
> >>> The RC tag in git is: release-2.7.3-RC1
> >>>
> >>> The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at https://repository.apache.org/
> content/repositories/orgapachehadoop-1045/ <https://repository.apache.
> org/content/repositories/orgapachehadoop-1045/>
> >>>
> >>> The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> for your quick perusal.
> >>>
> >>> As you may have noted,
> >>> - few issues with RC0 forced a RC1 [1]
> >>> - a very long fix-cycle for the License & Notice issues
> (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip
> by quite a bit. This release's related discussion thread is linked below:
> [2].
> >>>
> >>> Please try the release and vote; the vote will run for the usual 5
> days.
> >>>
> >>> Thanks,
> >>> Vinod
> >>>
> >>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0:
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/
> index.html#26106>
> >>> [2]: 2.7.3 release plan: https://www.mail-archive.com/
> hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/
> 6yv2fyrs4jlepmmr>
> >>
> >>
> >> ------------------------------------------------------------
> ---------
> >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
> > ------------------------------------------------------------
> ---------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
> > ------------------------------------------------------------
> ---------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Andrew Purtell <an...@gmail.com>.
>
What is a realistic strategy for us to evolve the HDFS audit log in a
backward-compatible way? If the API is essentially any form of ad-hoc
scripting, then for any proposed audit log format change, I can find a
reason to veto it on grounds of backward incompatibility.
Yeah when log scraping is the only way at information, then the API surface
expands to cover all manner of ad-hoc scripting.
Not sure moving away from emitting audit information in log lines would be
operator friendly. That's a tough one. Just about everything in the
ecosystem emits audit information as log lines. If Hadoop switches strategy
to become a one-off doing something different this would be painful.
Assuming log lines will be the way we continue to receive audit events from
Hadoop/HDFS, please consider freezing any changes to audit logging today,
develop a formal specification, add the specification to documentation, and
then take care to not break the specification between releases. Because
audit logging from the NN comes from low level places in FSNameSystem this
is going to constrain maintenance and refactor of that and related code, so
with my software maintainer hat on I feel your pain in advance. You'll want
to hash out what level of compatibility you'd like to offer. I'd recommend
only changing on major releases.
On Thu, Aug 18, 2016 at 10:04 AM, Chris Nauroth <cn...@hortonworks.com>
wrote:
> Andrew, thanks for adding your perspective on this.
>
>
> What is a realistic strategy for us to evolve the HDFS audit log in a
> backward-compatible way? If the API is essentially any form of ad-hoc
> scripting, then for any proposed audit log format change, I can find a
> reason to veto it on grounds of backward incompatibility.
>
> - I can’t add a new field on the end, because that would break an awk
> script that uses $NF expecting to find a specific field.
> - I can’t prepend a new field, because that would break a "cut -f1"
> expecting to find the timestamp.
> - HDFS can’t add any new features, because someone might have written a
> script that does "exit 1" if it finds an unexpected RPC in the "cmd=" field.
> - Hadoop is not allowed to add full IPv6 support, because someone might
> have written a script that looks at the "ip=" field and parses it by IPv4
> syntax.
>
> On the CLI, a potential solution for evolving the output is to preserve
> the old format by default and only enable the new format if the user
> explicitly passes a new argument. What should we do for the audit log?
> Configuration flags in hdfs-site.xml? (That of course adds its own brand
> of complexity.)
>
>
> I’m particularly interested to hear potential solutions from people like
> Andrew and Allen who have been most vocal about the need for a stable
> format. Without a solution, this unfortunately devolves into the format
> being frozen within a major release line.
>
> We could benefit from getting a patch on the compatibility doc that
> addresses the HDFS audit log specifically.
>
> --Chris Nauroth
>
> On 8/18/16, 8:47 AM, "Andrew Purtell" <an...@gmail.com> wrote:
>
> An incompatible APIs change is developer unfriendly. An incompatible
> behavioral change is operator unfriendly. Historically, one dimension of
> incompatibility has had a lot more mindshare than the other. It's great
> that this might be changing for the better.
>
> Where I work when we move from one Hadoop 2.x minor to another we
> always spend time updating our deployment plans, alerting, log scraping,
> and related things due to changes. Some are debatable as if qualifying for
> the 'incompatible' designation. I think the audit logging change that
> triggered this discussion is a good example of one that does. If you want
> to audit HDFS actions those log emissions are your API. (Inotify doesn't
> offer access control events.) One has to code regular expressions for
> parsing them and reverse engineer under what circumstances an audit line is
> emitted so you can make assumptions about what transpired. Change either
> and you might break someone's automation for meeting industry or legal
> compliance obligations. Not a trivial matter. If you don't operate Hadoop
> in production you might not realize the implications of such a change. Glad
> to see Hadoop has community diversity to recognize it in some cases.
>
> > On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
> >
> > I think Allen's previous comments are very misleading.
> > In my understanding, only incompatible API (RPC, CLIs, WebService,
> etc.) shouldn't land on branch-2, but other incompatible behaviors (logs,
> audit-log, daemon's restart, etc.) should get flexible for landing.
> Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with
> incompatible-changes could get landed on branch-2 after 2.2.0 release? Most
> of them are already released.
> >
> > Thanks,
> >
> > Junping
> > ________________________________________
> > From: Vinod Kumar Vavilapalli <vi...@apache.org>
> > Sent: Wednesday, August 17, 2016 9:29 PM
> > To: Allen Wittenauer
> > Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
> >
> > I always look at CHANGES.txt entries for incompatible-changes and
> this JIRA obviously wasn’t there.
> >
> > Anyways, this shouldn’t be in any of branch-2.* as committers there
> clearly mentioned that this is an incompatible change.
> >
> > I am reverting the patch from branch-2* .
> >
> > Thanks
> > +Vinod
> >
> >> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <
> aw@effectivemachines.com> wrote:
> >>
> >>
> >>
> >> -1
> >>
> >> HDFS-9395 is an incompatible change:
> >>
> >> a) Why is not marked as such in the changes file?
> >> b) Why is an incompatible change in a micro release, much less a
> minor?
> >> c) Where is the release note for this change?
> >>
> >>
> >>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
> >>>
> >>> As discussed before, this is the next maintenance release to
> follow up 2.7.2.
> >>>
> >>> The RC is available for validation at: http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC0/>
> >>>
> >>> The RC tag in git is: release-2.7.3-RC1
> >>>
> >>> The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at https://repository.apache.org/
> content/repositories/orgapachehadoop-1045/ <https://repository.apache.
> org/content/repositories/orgapachehadoop-1045/>
> >>>
> >>> The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> for your quick perusal.
> >>>
> >>> As you may have noted,
> >>> - few issues with RC0 forced a RC1 [1]
> >>> - a very long fix-cycle for the License & Notice issues
> (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip
> by quite a bit. This release's related discussion thread is linked below:
> [2].
> >>>
> >>> Please try the release and vote; the vote will run for the usual 5
> days.
> >>>
> >>> Thanks,
> >>> Vinod
> >>>
> >>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0:
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/
> index.html#26106>
> >>> [2]: 2.7.3 release plan: https://www.mail-archive.com/
> hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/
> 6yv2fyrs4jlepmmr>
> >>
> >>
> >> ------------------------------------------------------------
> ---------
> >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
> > ------------------------------------------------------------
> ---------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
> > ------------------------------------------------------------
> ---------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Kuhu Shukla <ks...@yahoo-inc.com.INVALID>.
Hi All,
Thank you for all the inputs on HDFS-9395. I have opened HDFS-10776 to discuss the modifications needed for audit logging to be consistent and comprehensive. We can move this discussion to the new JIRA.
Appreciate the support.
Regards,Kuhu Shukla
On Thursday, August 18, 2016 12:04 PM, Chris Nauroth <cn...@hortonworks.com> wrote:
Andrew, thanks for adding your perspective on this.
What is a realistic strategy for us to evolve the HDFS audit log in a backward-compatible way? If the API is essentially any form of ad-hoc scripting, then for any proposed audit log format change, I can find a reason to veto it on grounds of backward incompatibility.
- I can’t add a new field on the end, because that would break an awk script that uses $NF expecting to find a specific field.
- I can’t prepend a new field, because that would break a "cut -f1" expecting to find the timestamp.
- HDFS can’t add any new features, because someone might have written a script that does "exit 1" if it finds an unexpected RPC in the "cmd=" field.
- Hadoop is not allowed to add full IPv6 support, because someone might have written a script that looks at the "ip=" field and parses it by IPv4 syntax.
On the CLI, a potential solution for evolving the output is to preserve the old format by default and only enable the new format if the user explicitly passes a new argument. What should we do for the audit log? Configuration flags in hdfs-site.xml? (That of course adds its own brand of complexity.)
I’m particularly interested to hear potential solutions from people like Andrew and Allen who have been most vocal about the need for a stable format. Without a solution, this unfortunately devolves into the format being frozen within a major release line.
We could benefit from getting a patch on the compatibility doc that addresses the HDFS audit log specifically.
--Chris Nauroth
On 8/18/16, 8:47 AM, "Andrew Purtell" <an...@gmail.com> wrote:
An incompatible APIs change is developer unfriendly. An incompatible behavioral change is operator unfriendly. Historically, one dimension of incompatibility has had a lot more mindshare than the other. It's great that this might be changing for the better.
Where I work when we move from one Hadoop 2.x minor to another we always spend time updating our deployment plans, alerting, log scraping, and related things due to changes. Some are debatable as if qualifying for the 'incompatible' designation. I think the audit logging change that triggered this discussion is a good example of one that does. If you want to audit HDFS actions those log emissions are your API. (Inotify doesn't offer access control events.) One has to code regular expressions for parsing them and reverse engineer under what circumstances an audit line is emitted so you can make assumptions about what transpired. Change either and you might break someone's automation for meeting industry or legal compliance obligations. Not a trivial matter. If you don't operate Hadoop in production you might not realize the implications of such a change. Glad to see Hadoop has community diversity to recognize it in some cases.
> On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
> ________________________________________
> From: Vinod Kumar Vavilapalli <vi...@apache.org>
> Sent: Wednesday, August 17, 2016 9:29 PM
> To: Allen Wittenauer
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
>
> Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
>
> I am reverting the patch from branch-2* .
>
> Thanks
> +Vinod
>
>> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>>
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Kuhu Shukla <ks...@yahoo-inc.com.INVALID>.
Hi All,
Thank you for all the inputs on HDFS-9395. I have opened HDFS-10776 to discuss the modifications needed for audit logging to be consistent and comprehensive. We can move this discussion to the new JIRA.
Appreciate the support.
Regards,Kuhu Shukla
On Thursday, August 18, 2016 12:04 PM, Chris Nauroth <cn...@hortonworks.com> wrote:
Andrew, thanks for adding your perspective on this.
What is a realistic strategy for us to evolve the HDFS audit log in a backward-compatible way? If the API is essentially any form of ad-hoc scripting, then for any proposed audit log format change, I can find a reason to veto it on grounds of backward incompatibility.
- I can’t add a new field on the end, because that would break an awk script that uses $NF expecting to find a specific field.
- I can’t prepend a new field, because that would break a "cut -f1" expecting to find the timestamp.
- HDFS can’t add any new features, because someone might have written a script that does "exit 1" if it finds an unexpected RPC in the "cmd=" field.
- Hadoop is not allowed to add full IPv6 support, because someone might have written a script that looks at the "ip=" field and parses it by IPv4 syntax.
On the CLI, a potential solution for evolving the output is to preserve the old format by default and only enable the new format if the user explicitly passes a new argument. What should we do for the audit log? Configuration flags in hdfs-site.xml? (That of course adds its own brand of complexity.)
I’m particularly interested to hear potential solutions from people like Andrew and Allen who have been most vocal about the need for a stable format. Without a solution, this unfortunately devolves into the format being frozen within a major release line.
We could benefit from getting a patch on the compatibility doc that addresses the HDFS audit log specifically.
--Chris Nauroth
On 8/18/16, 8:47 AM, "Andrew Purtell" <an...@gmail.com> wrote:
An incompatible APIs change is developer unfriendly. An incompatible behavioral change is operator unfriendly. Historically, one dimension of incompatibility has had a lot more mindshare than the other. It's great that this might be changing for the better.
Where I work when we move from one Hadoop 2.x minor to another we always spend time updating our deployment plans, alerting, log scraping, and related things due to changes. Some are debatable as if qualifying for the 'incompatible' designation. I think the audit logging change that triggered this discussion is a good example of one that does. If you want to audit HDFS actions those log emissions are your API. (Inotify doesn't offer access control events.) One has to code regular expressions for parsing them and reverse engineer under what circumstances an audit line is emitted so you can make assumptions about what transpired. Change either and you might break someone's automation for meeting industry or legal compliance obligations. Not a trivial matter. If you don't operate Hadoop in production you might not realize the implications of such a change. Glad to see Hadoop has community diversity to recognize it in some cases.
> On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
> ________________________________________
> From: Vinod Kumar Vavilapalli <vi...@apache.org>
> Sent: Wednesday, August 17, 2016 9:29 PM
> To: Allen Wittenauer
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
>
> Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
>
> I am reverting the patch from branch-2* .
>
> Thanks
> +Vinod
>
>> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>>
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Andrew Purtell <an...@gmail.com>.
>
What is a realistic strategy for us to evolve the HDFS audit log in a
backward-compatible way? If the API is essentially any form of ad-hoc
scripting, then for any proposed audit log format change, I can find a
reason to veto it on grounds of backward incompatibility.
Yeah when log scraping is the only way at information, then the API surface
expands to cover all manner of ad-hoc scripting.
Not sure moving away from emitting audit information in log lines would be
operator friendly. That's a tough one. Just about everything in the
ecosystem emits audit information as log lines. If Hadoop switches strategy
to become a one-off doing something different this would be painful.
Assuming log lines will be the way we continue to receive audit events from
Hadoop/HDFS, please consider freezing any changes to audit logging today,
develop a formal specification, add the specification to documentation, and
then take care to not break the specification between releases. Because
audit logging from the NN comes from low level places in FSNameSystem this
is going to constrain maintenance and refactor of that and related code, so
with my software maintainer hat on I feel your pain in advance. You'll want
to hash out what level of compatibility you'd like to offer. I'd recommend
only changing on major releases.
On Thu, Aug 18, 2016 at 10:04 AM, Chris Nauroth <cn...@hortonworks.com>
wrote:
> Andrew, thanks for adding your perspective on this.
>
>
> What is a realistic strategy for us to evolve the HDFS audit log in a
> backward-compatible way? If the API is essentially any form of ad-hoc
> scripting, then for any proposed audit log format change, I can find a
> reason to veto it on grounds of backward incompatibility.
>
> - I can’t add a new field on the end, because that would break an awk
> script that uses $NF expecting to find a specific field.
> - I can’t prepend a new field, because that would break a "cut -f1"
> expecting to find the timestamp.
> - HDFS can’t add any new features, because someone might have written a
> script that does "exit 1" if it finds an unexpected RPC in the "cmd=" field.
> - Hadoop is not allowed to add full IPv6 support, because someone might
> have written a script that looks at the "ip=" field and parses it by IPv4
> syntax.
>
> On the CLI, a potential solution for evolving the output is to preserve
> the old format by default and only enable the new format if the user
> explicitly passes a new argument. What should we do for the audit log?
> Configuration flags in hdfs-site.xml? (That of course adds its own brand
> of complexity.)
>
>
> I’m particularly interested to hear potential solutions from people like
> Andrew and Allen who have been most vocal about the need for a stable
> format. Without a solution, this unfortunately devolves into the format
> being frozen within a major release line.
>
> We could benefit from getting a patch on the compatibility doc that
> addresses the HDFS audit log specifically.
>
> --Chris Nauroth
>
> On 8/18/16, 8:47 AM, "Andrew Purtell" <an...@gmail.com> wrote:
>
> An incompatible APIs change is developer unfriendly. An incompatible
> behavioral change is operator unfriendly. Historically, one dimension of
> incompatibility has had a lot more mindshare than the other. It's great
> that this might be changing for the better.
>
> Where I work when we move from one Hadoop 2.x minor to another we
> always spend time updating our deployment plans, alerting, log scraping,
> and related things due to changes. Some are debatable as if qualifying for
> the 'incompatible' designation. I think the audit logging change that
> triggered this discussion is a good example of one that does. If you want
> to audit HDFS actions those log emissions are your API. (Inotify doesn't
> offer access control events.) One has to code regular expressions for
> parsing them and reverse engineer under what circumstances an audit line is
> emitted so you can make assumptions about what transpired. Change either
> and you might break someone's automation for meeting industry or legal
> compliance obligations. Not a trivial matter. If you don't operate Hadoop
> in production you might not realize the implications of such a change. Glad
> to see Hadoop has community diversity to recognize it in some cases.
>
> > On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
> >
> > I think Allen's previous comments are very misleading.
> > In my understanding, only incompatible API (RPC, CLIs, WebService,
> etc.) shouldn't land on branch-2, but other incompatible behaviors (logs,
> audit-log, daemon's restart, etc.) should get flexible for landing.
> Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with
> incompatible-changes could get landed on branch-2 after 2.2.0 release? Most
> of them are already released.
> >
> > Thanks,
> >
> > Junping
> > ________________________________________
> > From: Vinod Kumar Vavilapalli <vi...@apache.org>
> > Sent: Wednesday, August 17, 2016 9:29 PM
> > To: Allen Wittenauer
> > Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
> >
> > I always look at CHANGES.txt entries for incompatible-changes and
> this JIRA obviously wasn’t there.
> >
> > Anyways, this shouldn’t be in any of branch-2.* as committers there
> clearly mentioned that this is an incompatible change.
> >
> > I am reverting the patch from branch-2* .
> >
> > Thanks
> > +Vinod
> >
> >> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <
> aw@effectivemachines.com> wrote:
> >>
> >>
> >>
> >> -1
> >>
> >> HDFS-9395 is an incompatible change:
> >>
> >> a) Why is not marked as such in the changes file?
> >> b) Why is an incompatible change in a micro release, much less a
> minor?
> >> c) Where is the release note for this change?
> >>
> >>
> >>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
> >>>
> >>> As discussed before, this is the next maintenance release to
> follow up 2.7.2.
> >>>
> >>> The RC is available for validation at: http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC0/>
> >>>
> >>> The RC tag in git is: release-2.7.3-RC1
> >>>
> >>> The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at https://repository.apache.org/
> content/repositories/orgapachehadoop-1045/ <https://repository.apache.
> org/content/repositories/orgapachehadoop-1045/>
> >>>
> >>> The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> for your quick perusal.
> >>>
> >>> As you may have noted,
> >>> - few issues with RC0 forced a RC1 [1]
> >>> - a very long fix-cycle for the License & Notice issues
> (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip
> by quite a bit. This release's related discussion thread is linked below:
> [2].
> >>>
> >>> Please try the release and vote; the vote will run for the usual 5
> days.
> >>>
> >>> Thanks,
> >>> Vinod
> >>>
> >>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0:
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/
> index.html#26106>
> >>> [2]: 2.7.3 release plan: https://www.mail-archive.com/
> hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/
> 6yv2fyrs4jlepmmr>
> >>
> >>
> >> ------------------------------------------------------------
> ---------
> >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
> > ------------------------------------------------------------
> ---------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
> > ------------------------------------------------------------
> ---------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Kuhu Shukla <ks...@yahoo-inc.com.INVALID>.
Hi All,
Thank you for all the inputs on HDFS-9395. I have opened HDFS-10776 to discuss the modifications needed for audit logging to be consistent and comprehensive. We can move this discussion to the new JIRA.
Appreciate the support.
Regards,Kuhu Shukla
On Thursday, August 18, 2016 12:04 PM, Chris Nauroth <cn...@hortonworks.com> wrote:
Andrew, thanks for adding your perspective on this.
What is a realistic strategy for us to evolve the HDFS audit log in a backward-compatible way? If the API is essentially any form of ad-hoc scripting, then for any proposed audit log format change, I can find a reason to veto it on grounds of backward incompatibility.
- I can’t add a new field on the end, because that would break an awk script that uses $NF expecting to find a specific field.
- I can’t prepend a new field, because that would break a "cut -f1" expecting to find the timestamp.
- HDFS can’t add any new features, because someone might have written a script that does "exit 1" if it finds an unexpected RPC in the "cmd=" field.
- Hadoop is not allowed to add full IPv6 support, because someone might have written a script that looks at the "ip=" field and parses it by IPv4 syntax.
On the CLI, a potential solution for evolving the output is to preserve the old format by default and only enable the new format if the user explicitly passes a new argument. What should we do for the audit log? Configuration flags in hdfs-site.xml? (That of course adds its own brand of complexity.)
I’m particularly interested to hear potential solutions from people like Andrew and Allen who have been most vocal about the need for a stable format. Without a solution, this unfortunately devolves into the format being frozen within a major release line.
We could benefit from getting a patch on the compatibility doc that addresses the HDFS audit log specifically.
--Chris Nauroth
On 8/18/16, 8:47 AM, "Andrew Purtell" <an...@gmail.com> wrote:
An incompatible APIs change is developer unfriendly. An incompatible behavioral change is operator unfriendly. Historically, one dimension of incompatibility has had a lot more mindshare than the other. It's great that this might be changing for the better.
Where I work when we move from one Hadoop 2.x minor to another we always spend time updating our deployment plans, alerting, log scraping, and related things due to changes. Some are debatable as if qualifying for the 'incompatible' designation. I think the audit logging change that triggered this discussion is a good example of one that does. If you want to audit HDFS actions those log emissions are your API. (Inotify doesn't offer access control events.) One has to code regular expressions for parsing them and reverse engineer under what circumstances an audit line is emitted so you can make assumptions about what transpired. Change either and you might break someone's automation for meeting industry or legal compliance obligations. Not a trivial matter. If you don't operate Hadoop in production you might not realize the implications of such a change. Glad to see Hadoop has community diversity to recognize it in some cases.
> On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
> ________________________________________
> From: Vinod Kumar Vavilapalli <vi...@apache.org>
> Sent: Wednesday, August 17, 2016 9:29 PM
> To: Allen Wittenauer
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
>
> Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
>
> I am reverting the patch from branch-2* .
>
> Thanks
> +Vinod
>
>> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>>
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Kuhu Shukla <ks...@yahoo-inc.com.INVALID>.
Hi All,
Thank you for all the inputs on HDFS-9395. I have opened HDFS-10776 to discuss the modifications needed for audit logging to be consistent and comprehensive. We can move this discussion to the new JIRA.
Appreciate the support.
Regards,Kuhu Shukla
On Thursday, August 18, 2016 12:04 PM, Chris Nauroth <cn...@hortonworks.com> wrote:
Andrew, thanks for adding your perspective on this.
What is a realistic strategy for us to evolve the HDFS audit log in a backward-compatible way? If the API is essentially any form of ad-hoc scripting, then for any proposed audit log format change, I can find a reason to veto it on grounds of backward incompatibility.
- I can’t add a new field on the end, because that would break an awk script that uses $NF expecting to find a specific field.
- I can’t prepend a new field, because that would break a "cut -f1" expecting to find the timestamp.
- HDFS can’t add any new features, because someone might have written a script that does "exit 1" if it finds an unexpected RPC in the "cmd=" field.
- Hadoop is not allowed to add full IPv6 support, because someone might have written a script that looks at the "ip=" field and parses it by IPv4 syntax.
On the CLI, a potential solution for evolving the output is to preserve the old format by default and only enable the new format if the user explicitly passes a new argument. What should we do for the audit log? Configuration flags in hdfs-site.xml? (That of course adds its own brand of complexity.)
I’m particularly interested to hear potential solutions from people like Andrew and Allen who have been most vocal about the need for a stable format. Without a solution, this unfortunately devolves into the format being frozen within a major release line.
We could benefit from getting a patch on the compatibility doc that addresses the HDFS audit log specifically.
--Chris Nauroth
On 8/18/16, 8:47 AM, "Andrew Purtell" <an...@gmail.com> wrote:
An incompatible APIs change is developer unfriendly. An incompatible behavioral change is operator unfriendly. Historically, one dimension of incompatibility has had a lot more mindshare than the other. It's great that this might be changing for the better.
Where I work when we move from one Hadoop 2.x minor to another we always spend time updating our deployment plans, alerting, log scraping, and related things due to changes. Some are debatable as if qualifying for the 'incompatible' designation. I think the audit logging change that triggered this discussion is a good example of one that does. If you want to audit HDFS actions those log emissions are your API. (Inotify doesn't offer access control events.) One has to code regular expressions for parsing them and reverse engineer under what circumstances an audit line is emitted so you can make assumptions about what transpired. Change either and you might break someone's automation for meeting industry or legal compliance obligations. Not a trivial matter. If you don't operate Hadoop in production you might not realize the implications of such a change. Glad to see Hadoop has community diversity to recognize it in some cases.
> On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
> ________________________________________
> From: Vinod Kumar Vavilapalli <vi...@apache.org>
> Sent: Wednesday, August 17, 2016 9:29 PM
> To: Allen Wittenauer
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
>
> Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
>
> I am reverting the patch from branch-2* .
>
> Thanks
> +Vinod
>
>> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>>
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Andrew Purtell <an...@gmail.com>.
>
What is a realistic strategy for us to evolve the HDFS audit log in a
backward-compatible way? If the API is essentially any form of ad-hoc
scripting, then for any proposed audit log format change, I can find a
reason to veto it on grounds of backward incompatibility.
Yeah when log scraping is the only way at information, then the API surface
expands to cover all manner of ad-hoc scripting.
Not sure moving away from emitting audit information in log lines would be
operator friendly. That's a tough one. Just about everything in the
ecosystem emits audit information as log lines. If Hadoop switches strategy
to become a one-off doing something different this would be painful.
Assuming log lines will be the way we continue to receive audit events from
Hadoop/HDFS, please consider freezing any changes to audit logging today,
develop a formal specification, add the specification to documentation, and
then take care to not break the specification between releases. Because
audit logging from the NN comes from low level places in FSNameSystem this
is going to constrain maintenance and refactor of that and related code, so
with my software maintainer hat on I feel your pain in advance. You'll want
to hash out what level of compatibility you'd like to offer. I'd recommend
only changing on major releases.
On Thu, Aug 18, 2016 at 10:04 AM, Chris Nauroth <cn...@hortonworks.com>
wrote:
> Andrew, thanks for adding your perspective on this.
>
>
> What is a realistic strategy for us to evolve the HDFS audit log in a
> backward-compatible way? If the API is essentially any form of ad-hoc
> scripting, then for any proposed audit log format change, I can find a
> reason to veto it on grounds of backward incompatibility.
>
> - I can’t add a new field on the end, because that would break an awk
> script that uses $NF expecting to find a specific field.
> - I can’t prepend a new field, because that would break a "cut -f1"
> expecting to find the timestamp.
> - HDFS can’t add any new features, because someone might have written a
> script that does "exit 1" if it finds an unexpected RPC in the "cmd=" field.
> - Hadoop is not allowed to add full IPv6 support, because someone might
> have written a script that looks at the "ip=" field and parses it by IPv4
> syntax.
>
> On the CLI, a potential solution for evolving the output is to preserve
> the old format by default and only enable the new format if the user
> explicitly passes a new argument. What should we do for the audit log?
> Configuration flags in hdfs-site.xml? (That of course adds its own brand
> of complexity.)
>
>
> I’m particularly interested to hear potential solutions from people like
> Andrew and Allen who have been most vocal about the need for a stable
> format. Without a solution, this unfortunately devolves into the format
> being frozen within a major release line.
>
> We could benefit from getting a patch on the compatibility doc that
> addresses the HDFS audit log specifically.
>
> --Chris Nauroth
>
> On 8/18/16, 8:47 AM, "Andrew Purtell" <an...@gmail.com> wrote:
>
> An incompatible APIs change is developer unfriendly. An incompatible
> behavioral change is operator unfriendly. Historically, one dimension of
> incompatibility has had a lot more mindshare than the other. It's great
> that this might be changing for the better.
>
> Where I work when we move from one Hadoop 2.x minor to another we
> always spend time updating our deployment plans, alerting, log scraping,
> and related things due to changes. Some are debatable as if qualifying for
> the 'incompatible' designation. I think the audit logging change that
> triggered this discussion is a good example of one that does. If you want
> to audit HDFS actions those log emissions are your API. (Inotify doesn't
> offer access control events.) One has to code regular expressions for
> parsing them and reverse engineer under what circumstances an audit line is
> emitted so you can make assumptions about what transpired. Change either
> and you might break someone's automation for meeting industry or legal
> compliance obligations. Not a trivial matter. If you don't operate Hadoop
> in production you might not realize the implications of such a change. Glad
> to see Hadoop has community diversity to recognize it in some cases.
>
> > On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
> >
> > I think Allen's previous comments are very misleading.
> > In my understanding, only incompatible API (RPC, CLIs, WebService,
> etc.) shouldn't land on branch-2, but other incompatible behaviors (logs,
> audit-log, daemon's restart, etc.) should get flexible for landing.
> Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with
> incompatible-changes could get landed on branch-2 after 2.2.0 release? Most
> of them are already released.
> >
> > Thanks,
> >
> > Junping
> > ________________________________________
> > From: Vinod Kumar Vavilapalli <vi...@apache.org>
> > Sent: Wednesday, August 17, 2016 9:29 PM
> > To: Allen Wittenauer
> > Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
> >
> > I always look at CHANGES.txt entries for incompatible-changes and
> this JIRA obviously wasn’t there.
> >
> > Anyways, this shouldn’t be in any of branch-2.* as committers there
> clearly mentioned that this is an incompatible change.
> >
> > I am reverting the patch from branch-2* .
> >
> > Thanks
> > +Vinod
> >
> >> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <
> aw@effectivemachines.com> wrote:
> >>
> >>
> >>
> >> -1
> >>
> >> HDFS-9395 is an incompatible change:
> >>
> >> a) Why is not marked as such in the changes file?
> >> b) Why is an incompatible change in a micro release, much less a
> minor?
> >> c) Where is the release note for this change?
> >>
> >>
> >>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
> >>>
> >>> As discussed before, this is the next maintenance release to
> follow up 2.7.2.
> >>>
> >>> The RC is available for validation at: http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC0/>
> >>>
> >>> The RC tag in git is: release-2.7.3-RC1
> >>>
> >>> The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at https://repository.apache.org/
> content/repositories/orgapachehadoop-1045/ <https://repository.apache.
> org/content/repositories/orgapachehadoop-1045/>
> >>>
> >>> The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> for your quick perusal.
> >>>
> >>> As you may have noted,
> >>> - few issues with RC0 forced a RC1 [1]
> >>> - a very long fix-cycle for the License & Notice issues
> (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip
> by quite a bit. This release's related discussion thread is linked below:
> [2].
> >>>
> >>> Please try the release and vote; the vote will run for the usual 5
> days.
> >>>
> >>> Thanks,
> >>> Vinod
> >>>
> >>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0:
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/
> index.html#26106>
> >>> [2]: 2.7.3 release plan: https://www.mail-archive.com/
> hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/
> 6yv2fyrs4jlepmmr>
> >>
> >>
> >> ------------------------------------------------------------
> ---------
> >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
> > ------------------------------------------------------------
> ---------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
> > ------------------------------------------------------------
> ---------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
>
>
>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Chris Nauroth <cn...@hortonworks.com>.
Andrew, thanks for adding your perspective on this.
What is a realistic strategy for us to evolve the HDFS audit log in a backward-compatible way? If the API is essentially any form of ad-hoc scripting, then for any proposed audit log format change, I can find a reason to veto it on grounds of backward incompatibility.
- I can’t add a new field on the end, because that would break an awk script that uses $NF expecting to find a specific field.
- I can’t prepend a new field, because that would break a "cut -f1" expecting to find the timestamp.
- HDFS can’t add any new features, because someone might have written a script that does "exit 1" if it finds an unexpected RPC in the "cmd=" field.
- Hadoop is not allowed to add full IPv6 support, because someone might have written a script that looks at the "ip=" field and parses it by IPv4 syntax.
On the CLI, a potential solution for evolving the output is to preserve the old format by default and only enable the new format if the user explicitly passes a new argument. What should we do for the audit log? Configuration flags in hdfs-site.xml? (That of course adds its own brand of complexity.)
I’m particularly interested to hear potential solutions from people like Andrew and Allen who have been most vocal about the need for a stable format. Without a solution, this unfortunately devolves into the format being frozen within a major release line.
We could benefit from getting a patch on the compatibility doc that addresses the HDFS audit log specifically.
--Chris Nauroth
On 8/18/16, 8:47 AM, "Andrew Purtell" <an...@gmail.com> wrote:
An incompatible APIs change is developer unfriendly. An incompatible behavioral change is operator unfriendly. Historically, one dimension of incompatibility has had a lot more mindshare than the other. It's great that this might be changing for the better.
Where I work when we move from one Hadoop 2.x minor to another we always spend time updating our deployment plans, alerting, log scraping, and related things due to changes. Some are debatable as if qualifying for the 'incompatible' designation. I think the audit logging change that triggered this discussion is a good example of one that does. If you want to audit HDFS actions those log emissions are your API. (Inotify doesn't offer access control events.) One has to code regular expressions for parsing them and reverse engineer under what circumstances an audit line is emitted so you can make assumptions about what transpired. Change either and you might break someone's automation for meeting industry or legal compliance obligations. Not a trivial matter. If you don't operate Hadoop in production you might not realize the implications of such a change. Glad to see Hadoop has community diversity to recognize it in some cases.
> On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
> ________________________________________
> From: Vinod Kumar Vavilapalli <vi...@apache.org>
> Sent: Wednesday, August 17, 2016 9:29 PM
> To: Allen Wittenauer
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
>
> Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
>
> I am reverting the patch from branch-2* .
>
> Thanks
> +Vinod
>
>> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>>
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Chris Nauroth <cn...@hortonworks.com>.
Andrew, thanks for adding your perspective on this.
What is a realistic strategy for us to evolve the HDFS audit log in a backward-compatible way? If the API is essentially any form of ad-hoc scripting, then for any proposed audit log format change, I can find a reason to veto it on grounds of backward incompatibility.
- I can’t add a new field on the end, because that would break an awk script that uses $NF expecting to find a specific field.
- I can’t prepend a new field, because that would break a "cut -f1" expecting to find the timestamp.
- HDFS can’t add any new features, because someone might have written a script that does "exit 1" if it finds an unexpected RPC in the "cmd=" field.
- Hadoop is not allowed to add full IPv6 support, because someone might have written a script that looks at the "ip=" field and parses it by IPv4 syntax.
On the CLI, a potential solution for evolving the output is to preserve the old format by default and only enable the new format if the user explicitly passes a new argument. What should we do for the audit log? Configuration flags in hdfs-site.xml? (That of course adds its own brand of complexity.)
I’m particularly interested to hear potential solutions from people like Andrew and Allen who have been most vocal about the need for a stable format. Without a solution, this unfortunately devolves into the format being frozen within a major release line.
We could benefit from getting a patch on the compatibility doc that addresses the HDFS audit log specifically.
--Chris Nauroth
On 8/18/16, 8:47 AM, "Andrew Purtell" <an...@gmail.com> wrote:
An incompatible APIs change is developer unfriendly. An incompatible behavioral change is operator unfriendly. Historically, one dimension of incompatibility has had a lot more mindshare than the other. It's great that this might be changing for the better.
Where I work when we move from one Hadoop 2.x minor to another we always spend time updating our deployment plans, alerting, log scraping, and related things due to changes. Some are debatable as if qualifying for the 'incompatible' designation. I think the audit logging change that triggered this discussion is a good example of one that does. If you want to audit HDFS actions those log emissions are your API. (Inotify doesn't offer access control events.) One has to code regular expressions for parsing them and reverse engineer under what circumstances an audit line is emitted so you can make assumptions about what transpired. Change either and you might break someone's automation for meeting industry or legal compliance obligations. Not a trivial matter. If you don't operate Hadoop in production you might not realize the implications of such a change. Glad to see Hadoop has community diversity to recognize it in some cases.
> On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
> ________________________________________
> From: Vinod Kumar Vavilapalli <vi...@apache.org>
> Sent: Wednesday, August 17, 2016 9:29 PM
> To: Allen Wittenauer
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
>
> Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
>
> I am reverting the patch from branch-2* .
>
> Thanks
> +Vinod
>
>> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>>
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Chris Nauroth <cn...@hortonworks.com>.
Andrew, thanks for adding your perspective on this.
What is a realistic strategy for us to evolve the HDFS audit log in a backward-compatible way? If the API is essentially any form of ad-hoc scripting, then for any proposed audit log format change, I can find a reason to veto it on grounds of backward incompatibility.
- I can’t add a new field on the end, because that would break an awk script that uses $NF expecting to find a specific field.
- I can’t prepend a new field, because that would break a "cut -f1" expecting to find the timestamp.
- HDFS can’t add any new features, because someone might have written a script that does "exit 1" if it finds an unexpected RPC in the "cmd=" field.
- Hadoop is not allowed to add full IPv6 support, because someone might have written a script that looks at the "ip=" field and parses it by IPv4 syntax.
On the CLI, a potential solution for evolving the output is to preserve the old format by default and only enable the new format if the user explicitly passes a new argument. What should we do for the audit log? Configuration flags in hdfs-site.xml? (That of course adds its own brand of complexity.)
I’m particularly interested to hear potential solutions from people like Andrew and Allen who have been most vocal about the need for a stable format. Without a solution, this unfortunately devolves into the format being frozen within a major release line.
We could benefit from getting a patch on the compatibility doc that addresses the HDFS audit log specifically.
--Chris Nauroth
On 8/18/16, 8:47 AM, "Andrew Purtell" <an...@gmail.com> wrote:
An incompatible APIs change is developer unfriendly. An incompatible behavioral change is operator unfriendly. Historically, one dimension of incompatibility has had a lot more mindshare than the other. It's great that this might be changing for the better.
Where I work when we move from one Hadoop 2.x minor to another we always spend time updating our deployment plans, alerting, log scraping, and related things due to changes. Some are debatable as if qualifying for the 'incompatible' designation. I think the audit logging change that triggered this discussion is a good example of one that does. If you want to audit HDFS actions those log emissions are your API. (Inotify doesn't offer access control events.) One has to code regular expressions for parsing them and reverse engineer under what circumstances an audit line is emitted so you can make assumptions about what transpired. Change either and you might break someone's automation for meeting industry or legal compliance obligations. Not a trivial matter. If you don't operate Hadoop in production you might not realize the implications of such a change. Glad to see Hadoop has community diversity to recognize it in some cases.
> On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
> ________________________________________
> From: Vinod Kumar Vavilapalli <vi...@apache.org>
> Sent: Wednesday, August 17, 2016 9:29 PM
> To: Allen Wittenauer
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
>
> Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
>
> I am reverting the patch from branch-2* .
>
> Thanks
> +Vinod
>
>> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>>
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Chris Nauroth <cn...@hortonworks.com>.
Andrew, thanks for adding your perspective on this.
What is a realistic strategy for us to evolve the HDFS audit log in a backward-compatible way? If the API is essentially any form of ad-hoc scripting, then for any proposed audit log format change, I can find a reason to veto it on grounds of backward incompatibility.
- I can’t add a new field on the end, because that would break an awk script that uses $NF expecting to find a specific field.
- I can’t prepend a new field, because that would break a "cut -f1" expecting to find the timestamp.
- HDFS can’t add any new features, because someone might have written a script that does "exit 1" if it finds an unexpected RPC in the "cmd=" field.
- Hadoop is not allowed to add full IPv6 support, because someone might have written a script that looks at the "ip=" field and parses it by IPv4 syntax.
On the CLI, a potential solution for evolving the output is to preserve the old format by default and only enable the new format if the user explicitly passes a new argument. What should we do for the audit log? Configuration flags in hdfs-site.xml? (That of course adds its own brand of complexity.)
I’m particularly interested to hear potential solutions from people like Andrew and Allen who have been most vocal about the need for a stable format. Without a solution, this unfortunately devolves into the format being frozen within a major release line.
We could benefit from getting a patch on the compatibility doc that addresses the HDFS audit log specifically.
--Chris Nauroth
On 8/18/16, 8:47 AM, "Andrew Purtell" <an...@gmail.com> wrote:
An incompatible APIs change is developer unfriendly. An incompatible behavioral change is operator unfriendly. Historically, one dimension of incompatibility has had a lot more mindshare than the other. It's great that this might be changing for the better.
Where I work when we move from one Hadoop 2.x minor to another we always spend time updating our deployment plans, alerting, log scraping, and related things due to changes. Some are debatable as if qualifying for the 'incompatible' designation. I think the audit logging change that triggered this discussion is a good example of one that does. If you want to audit HDFS actions those log emissions are your API. (Inotify doesn't offer access control events.) One has to code regular expressions for parsing them and reverse engineer under what circumstances an audit line is emitted so you can make assumptions about what transpired. Change either and you might break someone's automation for meeting industry or legal compliance obligations. Not a trivial matter. If you don't operate Hadoop in production you might not realize the implications of such a change. Glad to see Hadoop has community diversity to recognize it in some cases.
> On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
> ________________________________________
> From: Vinod Kumar Vavilapalli <vi...@apache.org>
> Sent: Wednesday, August 17, 2016 9:29 PM
> To: Allen Wittenauer
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
>
> Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
>
> I am reverting the patch from branch-2* .
>
> Thanks
> +Vinod
>
>> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>>
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Andrew Purtell <an...@gmail.com>.
An incompatible APIs change is developer unfriendly. An incompatible behavioral change is operator unfriendly. Historically, one dimension of incompatibility has had a lot more mindshare than the other. It's great that this might be changing for the better.
Where I work when we move from one Hadoop 2.x minor to another we always spend time updating our deployment plans, alerting, log scraping, and related things due to changes. Some are debatable as if qualifying for the 'incompatible' designation. I think the audit logging change that triggered this discussion is a good example of one that does. If you want to audit HDFS actions those log emissions are your API. (Inotify doesn't offer access control events.) One has to code regular expressions for parsing them and reverse engineer under what circumstances an audit line is emitted so you can make assumptions about what transpired. Change either and you might break someone's automation for meeting industry or legal compliance obligations. Not a trivial matter. If you don't operate Hadoop in production you might not realize the implications of such a change. Glad to see Hadoop has community diversity to recognize it in some cases.
> On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
> ________________________________________
> From: Vinod Kumar Vavilapalli <vi...@apache.org>
> Sent: Wednesday, August 17, 2016 9:29 PM
> To: Allen Wittenauer
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
>
> Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
>
> I am reverting the patch from branch-2* .
>
> Thanks
> +Vinod
>
>> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>>
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Andrew Purtell <an...@gmail.com>.
An incompatible APIs change is developer unfriendly. An incompatible behavioral change is operator unfriendly. Historically, one dimension of incompatibility has had a lot more mindshare than the other. It's great that this might be changing for the better.
Where I work when we move from one Hadoop 2.x minor to another we always spend time updating our deployment plans, alerting, log scraping, and related things due to changes. Some are debatable as if qualifying for the 'incompatible' designation. I think the audit logging change that triggered this discussion is a good example of one that does. If you want to audit HDFS actions those log emissions are your API. (Inotify doesn't offer access control events.) One has to code regular expressions for parsing them and reverse engineer under what circumstances an audit line is emitted so you can make assumptions about what transpired. Change either and you might break someone's automation for meeting industry or legal compliance obligations. Not a trivial matter. If you don't operate Hadoop in production you might not realize the implications of such a change. Glad to see Hadoop has community diversity to recognize it in some cases.
> On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
> ________________________________________
> From: Vinod Kumar Vavilapalli <vi...@apache.org>
> Sent: Wednesday, August 17, 2016 9:29 PM
> To: Allen Wittenauer
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
>
> Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
>
> I am reverting the patch from branch-2* .
>
> Thanks
> +Vinod
>
>> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>>
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Steve Loughran <st...@hortonworks.com>.
> On 18 Aug 2016, at 14:57, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
Don't get AW started on compatiblity; it'll only upset him.
One thing he does care about is the ability of programs to consume the output of commands and logs —and for that even the output of commands and logs need to continue to be parseable
https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/Compatibility.html#Command_Line_Interface_CLI
" Changing the path of a command, removing or renaming command line options, the order of arguments, or the command return code and output break compatibility and may adversely affect users."
I believe Allen is particularly concerned that a minor point release is going in as incompatible, on the basis the audit log output will change —that's the log that is explicitly designed for machine processing, hooking up to flume & kafka, etc. As example, Spotify spoke at a Hadoop Summit conference about how they used it to identify files which hadn't been used for a long time; inferring an atime attribute from the access history.
What has changed in the output?
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Steve Loughran <st...@hortonworks.com>.
> On 18 Aug 2016, at 14:57, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
Don't get AW started on compatiblity; it'll only upset him.
One thing he does care about is the ability of programs to consume the output of commands and logs —and for that even the output of commands and logs need to continue to be parseable
https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/Compatibility.html#Command_Line_Interface_CLI
" Changing the path of a command, removing or renaming command line options, the order of arguments, or the command return code and output break compatibility and may adversely affect users."
I believe Allen is particularly concerned that a minor point release is going in as incompatible, on the basis the audit log output will change —that's the log that is explicitly designed for machine processing, hooking up to flume & kafka, etc. As example, Spotify spoke at a Hadoop Summit conference about how they used it to identify files which hadn't been used for a long time; inferring an atime attribute from the access history.
What has changed in the output?
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Andrew Purtell <an...@gmail.com>.
An incompatible APIs change is developer unfriendly. An incompatible behavioral change is operator unfriendly. Historically, one dimension of incompatibility has had a lot more mindshare than the other. It's great that this might be changing for the better.
Where I work when we move from one Hadoop 2.x minor to another we always spend time updating our deployment plans, alerting, log scraping, and related things due to changes. Some are debatable as if qualifying for the 'incompatible' designation. I think the audit logging change that triggered this discussion is a good example of one that does. If you want to audit HDFS actions those log emissions are your API. (Inotify doesn't offer access control events.) One has to code regular expressions for parsing them and reverse engineer under what circumstances an audit line is emitted so you can make assumptions about what transpired. Change either and you might break someone's automation for meeting industry or legal compliance obligations. Not a trivial matter. If you don't operate Hadoop in production you might not realize the implications of such a change. Glad to see Hadoop has community diversity to recognize it in some cases.
> On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
> ________________________________________
> From: Vinod Kumar Vavilapalli <vi...@apache.org>
> Sent: Wednesday, August 17, 2016 9:29 PM
> To: Allen Wittenauer
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
>
> Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
>
> I am reverting the patch from branch-2* .
>
> Thanks
> +Vinod
>
>> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>>
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Andrew Purtell <an...@gmail.com>.
An incompatible APIs change is developer unfriendly. An incompatible behavioral change is operator unfriendly. Historically, one dimension of incompatibility has had a lot more mindshare than the other. It's great that this might be changing for the better.
Where I work when we move from one Hadoop 2.x minor to another we always spend time updating our deployment plans, alerting, log scraping, and related things due to changes. Some are debatable as if qualifying for the 'incompatible' designation. I think the audit logging change that triggered this discussion is a good example of one that does. If you want to audit HDFS actions those log emissions are your API. (Inotify doesn't offer access control events.) One has to code regular expressions for parsing them and reverse engineer under what circumstances an audit line is emitted so you can make assumptions about what transpired. Change either and you might break someone's automation for meeting industry or legal compliance obligations. Not a trivial matter. If you don't operate Hadoop in production you might not realize the implications of such a change. Glad to see Hadoop has community diversity to recognize it in some cases.
> On Aug 18, 2016, at 6:57 AM, Junping Du <jd...@hortonworks.com> wrote:
>
> I think Allen's previous comments are very misleading.
> In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
>
> Thanks,
>
> Junping
> ________________________________________
> From: Vinod Kumar Vavilapalli <vi...@apache.org>
> Sent: Wednesday, August 17, 2016 9:29 PM
> To: Allen Wittenauer
> Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
>
> I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
>
> Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
>
> I am reverting the patch from branch-2* .
>
> Thanks
> +Vinod
>
>> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>>
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Junping Du <jd...@hortonworks.com>.
I think Allen's previous comments are very misleading.
In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
Thanks,
Junping
________________________________________
From: Vinod Kumar Vavilapalli <vi...@apache.org>
Sent: Wednesday, August 17, 2016 9:29 PM
To: Allen Wittenauer
Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
I am reverting the patch from branch-2* .
Thanks
+Vinod
> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Junping Du <jd...@hortonworks.com>.
I think Allen's previous comments are very misleading.
In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
Thanks,
Junping
________________________________________
From: Vinod Kumar Vavilapalli <vi...@apache.org>
Sent: Wednesday, August 17, 2016 9:29 PM
To: Allen Wittenauer
Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
I am reverting the patch from branch-2* .
Thanks
+Vinod
> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Junping Du <jd...@hortonworks.com>.
I think Allen's previous comments are very misleading.
In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
Thanks,
Junping
________________________________________
From: Vinod Kumar Vavilapalli <vi...@apache.org>
Sent: Wednesday, August 17, 2016 9:29 PM
To: Allen Wittenauer
Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
I am reverting the patch from branch-2* .
Thanks
+Vinod
> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Junping Du <jd...@hortonworks.com>.
I think Allen's previous comments are very misleading.
In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released.
Thanks,
Junping
________________________________________
From: Vinod Kumar Vavilapalli <vi...@apache.org>
Sent: Wednesday, August 17, 2016 9:29 PM
To: Allen Wittenauer
Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
I am reverting the patch from branch-2* .
Thanks
+Vinod
> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
I am reverting the patch from branch-2* .
Thanks
+Vinod
> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
I am reverting the patch from branch-2* .
Thanks
+Vinod
> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Junping Du <jd...@hortonworks.com>.
From my quick understanding, HDFS-9395 is more like a bug fix and improvement for audit logging instead of incompatible changes. We mark incompatible probably because the audit log behavior could be corrected/updated in some exception cases. I think it still belongs to 2.7.3 scope.
Kuhu and Kihwal, any comments here?
Thanks,
Junping
________________________________________
From: Allen Wittenauer <aw...@effectivemachines.com>
Sent: Wednesday, August 17, 2016 5:29 AM
To: common-dev@hadoop.apache.org
Cc: hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
-1
HDFS-9395 is an incompatible change:
a) Why is not marked as such in the changes file?
b) Why is an incompatible change in a micro release, much less a minor?
c) Where is the release note for this change?
> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Canceling the release vote for this and other issues reported.
+Vinod
> On Aug 16, 2016, at 10:01 PM, Akira Ajisaka <aj...@oss.nttdata.co.jp> wrote:
>
> -1 (binding)
>
> HADOOP-13434 and HADOOP-11814, committed between RC0 and RC1, are not reflected in the release note.
>
> -Akira
>
> On 8/17/16 13:29, Allen Wittenauer wrote:
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Canceling the release vote for this and other issues reported.
+Vinod
> On Aug 16, 2016, at 10:01 PM, Akira Ajisaka <aj...@oss.nttdata.co.jp> wrote:
>
> -1 (binding)
>
> HADOOP-13434 and HADOOP-11814, committed between RC0 and RC1, are not reflected in the release note.
>
> -Akira
>
> On 8/17/16 13:29, Allen Wittenauer wrote:
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Canceling the release vote for this and other issues reported.
+Vinod
> On Aug 16, 2016, at 10:01 PM, Akira Ajisaka <aj...@oss.nttdata.co.jp> wrote:
>
> -1 (binding)
>
> HADOOP-13434 and HADOOP-11814, committed between RC0 and RC1, are not reflected in the release note.
>
> -Akira
>
> On 8/17/16 13:29, Allen Wittenauer wrote:
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Canceling the release vote for this and other issues reported.
+Vinod
> On Aug 16, 2016, at 10:01 PM, Akira Ajisaka <aj...@oss.nttdata.co.jp> wrote:
>
> -1 (binding)
>
> HADOOP-13434 and HADOOP-11814, committed between RC0 and RC1, are not reflected in the release note.
>
> -Akira
>
> On 8/17/16 13:29, Allen Wittenauer wrote:
>>
>>
>> -1
>>
>> HDFS-9395 is an incompatible change:
>>
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>>
>>
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>>
>>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>>
>>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>>
>>> The RC tag in git is: release-2.7.3-RC1
>>>
>>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>>
>>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>>
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>>
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>>
>>> Thanks,
>>> Vinod
>>>
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Akira Ajisaka <aj...@oss.nttdata.co.jp>.
-1 (binding)
HADOOP-13434 and HADOOP-11814, committed between RC0 and RC1, are not
reflected in the release note.
-Akira
On 8/17/16 13:29, Allen Wittenauer wrote:
>
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Akira Ajisaka <aj...@oss.nttdata.co.jp>.
-1 (binding)
HADOOP-13434 and HADOOP-11814, committed between RC0 and RC1, are not
reflected in the release note.
-Akira
On 8/17/16 13:29, Allen Wittenauer wrote:
>
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Akira Ajisaka <aj...@oss.nttdata.co.jp>.
-1 (binding)
HADOOP-13434 and HADOOP-11814, committed between RC0 and RC1, are not
reflected in the release note.
-Akira
On 8/17/16 13:29, Allen Wittenauer wrote:
>
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Junping Du <jd...@hortonworks.com>.
From my quick understanding, HDFS-9395 is more like a bug fix and improvement for audit logging instead of incompatible changes. We mark incompatible probably because the audit log behavior could be corrected/updated in some exception cases. I think it still belongs to 2.7.3 scope.
Kuhu and Kihwal, any comments here?
Thanks,
Junping
________________________________________
From: Allen Wittenauer <aw...@effectivemachines.com>
Sent: Wednesday, August 17, 2016 5:29 AM
To: common-dev@hadoop.apache.org
Cc: hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
-1
HDFS-9395 is an incompatible change:
a) Why is not marked as such in the changes file?
b) Why is an incompatible change in a micro release, much less a minor?
c) Where is the release note for this change?
> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Junping Du <jd...@hortonworks.com>.
From my quick understanding, HDFS-9395 is more like a bug fix and improvement for audit logging instead of incompatible changes. We mark incompatible probably because the audit log behavior could be corrected/updated in some exception cases. I think it still belongs to 2.7.3 scope.
Kuhu and Kihwal, any comments here?
Thanks,
Junping
________________________________________
From: Allen Wittenauer <aw...@effectivemachines.com>
Sent: Wednesday, August 17, 2016 5:29 AM
To: common-dev@hadoop.apache.org
Cc: hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
-1
HDFS-9395 is an incompatible change:
a) Why is not marked as such in the changes file?
b) Why is an incompatible change in a micro release, much less a minor?
c) Where is the release note for this change?
> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
I am reverting the patch from branch-2* .
Thanks
+Vinod
> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there.
Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change.
I am reverting the patch from branch-2* .
Thanks
+Vinod
> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Akira Ajisaka <aj...@oss.nttdata.co.jp>.
-1 (binding)
HADOOP-13434 and HADOOP-11814, committed between RC0 and RC1, are not
reflected in the release note.
-Akira
On 8/17/16 13:29, Allen Wittenauer wrote:
>
>
> -1
>
> HDFS-9395 is an incompatible change:
>
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
>
>
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>>
>> Hi all,
>>
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>>
>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC1
>>
>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>
>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>>
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
-1
HDFS-9395 is an incompatible change:
a) Why is not marked as such in the changes file?
b) Why is an incompatible change in a micro release, much less a minor?
c) Where is the release note for this change?
> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
-1
HDFS-9395 is an incompatible change:
a) Why is not marked as such in the changes file?
b) Why is an incompatible change in a micro release, much less a minor?
c) Where is the release note for this change?
> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Steve Loughran <st...@hortonworks.com>.
+1 binding
1. built and tested apache slider (incubating) against the Hadoop 2.7.3 artifacts
2. did a build & test of Apache Spark master branch iwth 2.7.3 JARs,
For that I had to tweak spark's build to support the staging repo; hopefully that will get into Spark
https://issues.apache.org/jira/browse/SPARK-17058
3. did a test run of my WiP SPARK-7481 spark-cloud module; after fixing a couple of things on the test setup side related to HADOOP-13058,
mvn test --pl cloud -Pyarn,hadoop-2.7,snapshots-and-staging -Dhadoop.version=2.7.3 -Dcloud.test.configuration.file=../conf/cloud-tests.xml
all was well —albeit measurably slower than Hadoop 2.8. That's proof that the 2.8 version of s3a really does deliver measurable speedup for those tests (currently just file input/seek; more to come). I had originally thought things were broken as s3 init was failing -but that's because the s3 bucket was in frankfurt, and the AWS library used can't talk to that endpoint (v4 auth protocol, see).
4. did a full spark distribution build of that SPARK-7481 branch
dev/make-distribution.sh -Pyarn,hadoop-2.7,snapshots-and-staging -Dhadoop.version=2.7.3
ran command line test to do read of s3a data:
bin/spark-submit --class org.apache.spark.cloud.s3.examples.S3LineCount \
--conf spark.hadoop.fs.s3a.access.key=$AWS_KEY \
--conf spark.hadoop.fs.s3a.secret.key=$AWS_SECRET \
examples/jars/spark-examples_2.11-2.1.0-SNAPSHOT.jar
5. Pulled out the microsoft Azure JAR azure-storage-2.0.0.jar and repeated step 4
-this showed that the 2.7.x branch does handle the failure to load a filesystem due to dependency or other classloading problems —this was proving a big problem in adding the aws & azure stuff to the spark build, as it'd stop spark from starting up if the dependencies were absent.
I've not done any of the .tar.gz diligence; I've just looked at the staged JARs and how they worked with downstream apps —that being a key way that Hadoop artifacts are adopted.
> On 12 Aug 2016, at 17:45, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
-1
HDFS-9395 is an incompatible change:
a) Why is not marked as such in the changes file?
b) Why is an incompatible change in a micro release, much less a minor?
c) Where is the release note for this change?
> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Marco Zühlke <mz...@gmail.com>.
Hi Vinod,
I'm not sure if this is relevant, but you changed the release date in the
CHANGES.txt
<https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f#diff-4de1a6452466a82b89570bd9ab606c12>
files to 2016-09-19.
I guess you have meant 2016-08-19.
See:
https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f
Thanks,
Marco
2016-08-12 18:45 GMT+02:00 Vinod Kumar Vavilapalli <vi...@apache.org>:
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up
> 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at https://repository.apache.org/
> content/repositories/orgapachehadoop-1045/ <https://repository.apache.
> org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893)
> caused 2.7.3 (along with every other Hadoop release) to slip by quite a
> bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/
> hdfs-dev%40hadoop.apache.org/index.html#26106 <
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106
> >
> [2]: 2.7.3 release plan: https://www.mail-archive.com/
> hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/
> 6yv2fyrs4jlepmmr>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Steve Loughran <st...@hortonworks.com>.
+1 binding
1. built and tested apache slider (incubating) against the Hadoop 2.7.3 artifacts
2. did a build & test of Apache Spark master branch iwth 2.7.3 JARs,
For that I had to tweak spark's build to support the staging repo; hopefully that will get into Spark
https://issues.apache.org/jira/browse/SPARK-17058
3. did a test run of my WiP SPARK-7481 spark-cloud module; after fixing a couple of things on the test setup side related to HADOOP-13058,
mvn test --pl cloud -Pyarn,hadoop-2.7,snapshots-and-staging -Dhadoop.version=2.7.3 -Dcloud.test.configuration.file=../conf/cloud-tests.xml
all was well —albeit measurably slower than Hadoop 2.8. That's proof that the 2.8 version of s3a really does deliver measurable speedup for those tests (currently just file input/seek; more to come). I had originally thought things were broken as s3 init was failing -but that's because the s3 bucket was in frankfurt, and the AWS library used can't talk to that endpoint (v4 auth protocol, see).
4. did a full spark distribution build of that SPARK-7481 branch
dev/make-distribution.sh -Pyarn,hadoop-2.7,snapshots-and-staging -Dhadoop.version=2.7.3
ran command line test to do read of s3a data:
bin/spark-submit --class org.apache.spark.cloud.s3.examples.S3LineCount \
--conf spark.hadoop.fs.s3a.access.key=$AWS_KEY \
--conf spark.hadoop.fs.s3a.secret.key=$AWS_SECRET \
examples/jars/spark-examples_2.11-2.1.0-SNAPSHOT.jar
5. Pulled out the microsoft Azure JAR azure-storage-2.0.0.jar and repeated step 4
-this showed that the 2.7.x branch does handle the failure to load a filesystem due to dependency or other classloading problems —this was proving a big problem in adding the aws & azure stuff to the spark build, as it'd stop spark from starting up if the dependencies were absent.
I've not done any of the .tar.gz diligence; I've just looked at the staged JARs and how they worked with downstream apps —that being a key way that Hadoop artifacts are adopted.
> On 12 Aug 2016, at 17:45, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
-1
HDFS-9395 is an incompatible change:
a) Why is not marked as such in the changes file?
b) Why is an incompatible change in a micro release, much less a minor?
c) Where is the release note for this change?
> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Jason Lowe <jl...@yahoo-inc.com.INVALID>.
+1 (binding)
- Verified signatures and digests- Built from source with native support- Deployed a pseudo-distributed cluster- Ran some sample jobs
Jason
From: Vinod Kumar Vavilapalli <vi...@apache.org>
To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
Cc: Vinod Kumar Vavilapalli <vi...@apache.org>
Sent: Friday, August 12, 2016 11:45 AM
Subject: [VOTE] Release Apache Hadoop 2.7.3 RC1
Hi all,
I've created a release candidate RC1 for Apache Hadoop 2.7.3.
As discussed before, this is the next maintenance release to follow up 2.7.2.
The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
The RC tag in git is: release-2.7.3-RC1
The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
As you may have noted,
- few issues with RC0 forced a RC1 [1]
- a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
Please try the release and vote; the vote will run for the usual 5 days.
Thanks,
Vinod
[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
[2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Marco Zühlke <mz...@gmail.com>.
Hi Vinod,
I'm not sure if this is relevant, but you changed the release date in the
CHANGES.txt
<https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f#diff-4de1a6452466a82b89570bd9ab606c12>
files to 2016-09-19.
I guess you have meant 2016-08-19.
See:
https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f
Thanks,
Marco
2016-08-12 18:45 GMT+02:00 Vinod Kumar Vavilapalli <vi...@apache.org>:
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up
> 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at https://repository.apache.org/
> content/repositories/orgapachehadoop-1045/ <https://repository.apache.
> org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893)
> caused 2.7.3 (along with every other Hadoop release) to slip by quite a
> bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/
> hdfs-dev%40hadoop.apache.org/index.html#26106 <
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106
> >
> [2]: 2.7.3 release plan: https://www.mail-archive.com/
> hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/
> 6yv2fyrs4jlepmmr>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Steve Loughran <st...@hortonworks.com>.
+1 binding
1. built and tested apache slider (incubating) against the Hadoop 2.7.3 artifacts
2. did a build & test of Apache Spark master branch iwth 2.7.3 JARs,
For that I had to tweak spark's build to support the staging repo; hopefully that will get into Spark
https://issues.apache.org/jira/browse/SPARK-17058
3. did a test run of my WiP SPARK-7481 spark-cloud module; after fixing a couple of things on the test setup side related to HADOOP-13058,
mvn test --pl cloud -Pyarn,hadoop-2.7,snapshots-and-staging -Dhadoop.version=2.7.3 -Dcloud.test.configuration.file=../conf/cloud-tests.xml
all was well —albeit measurably slower than Hadoop 2.8. That's proof that the 2.8 version of s3a really does deliver measurable speedup for those tests (currently just file input/seek; more to come). I had originally thought things were broken as s3 init was failing -but that's because the s3 bucket was in frankfurt, and the AWS library used can't talk to that endpoint (v4 auth protocol, see).
4. did a full spark distribution build of that SPARK-7481 branch
dev/make-distribution.sh -Pyarn,hadoop-2.7,snapshots-and-staging -Dhadoop.version=2.7.3
ran command line test to do read of s3a data:
bin/spark-submit --class org.apache.spark.cloud.s3.examples.S3LineCount \
--conf spark.hadoop.fs.s3a.access.key=$AWS_KEY \
--conf spark.hadoop.fs.s3a.secret.key=$AWS_SECRET \
examples/jars/spark-examples_2.11-2.1.0-SNAPSHOT.jar
5. Pulled out the microsoft Azure JAR azure-storage-2.0.0.jar and repeated step 4
-this showed that the 2.7.x branch does handle the failure to load a filesystem due to dependency or other classloading problems —this was proving a big problem in adding the aws & azure stuff to the spark build, as it'd stop spark from starting up if the dependencies were absent.
I've not done any of the .tar.gz diligence; I've just looked at the staged JARs and how they worked with downstream apps —that being a key way that Hadoop artifacts are adopted.
> On 12 Aug 2016, at 17:45, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Jason Lowe <jl...@yahoo-inc.com.INVALID>.
+1 (binding)
- Verified signatures and digests- Built from source with native support- Deployed a pseudo-distributed cluster- Ran some sample jobs
Jason
From: Vinod Kumar Vavilapalli <vi...@apache.org>
To: "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>
Cc: Vinod Kumar Vavilapalli <vi...@apache.org>
Sent: Friday, August 12, 2016 11:45 AM
Subject: [VOTE] Release Apache Hadoop 2.7.3 RC1
Hi all,
I've created a release candidate RC1 for Apache Hadoop 2.7.3.
As discussed before, this is the next maintenance release to follow up 2.7.2.
The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
The RC tag in git is: release-2.7.3-RC1
The maven artifacts are available via repository.apache.org <http://repository.apache.org/> at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your quick perusal.
As you may have noted,
- few issues with RC0 forced a RC1 [1]
- a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's related discussion thread is linked below: [2].
Please try the release and vote; the vote will run for the usual 5 days.
Thanks,
Vinod
[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
[2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/6yv2fyrs4jlepmmr>
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Posted by Marco Zühlke <mz...@gmail.com>.
Hi Vinod,
I'm not sure if this is relevant, but you changed the release date in the
CHANGES.txt
<https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f#diff-4de1a6452466a82b89570bd9ab606c12>
files to 2016-09-19.
I guess you have meant 2016-08-19.
See:
https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f
Thanks,
Marco
2016-08-12 18:45 GMT+02:00 Vinod Kumar Vavilapalli <vi...@apache.org>:
> Hi all,
>
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up
> 2.7.2.
>
> The RC is available for validation at: http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC1/ <http://home.apache.org/~
> vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC1
>
> The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at https://repository.apache.org/
> content/repositories/orgapachehadoop-1045/ <https://repository.apache.
> org/content/repositories/orgapachehadoop-1045/>
>
> The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> for your quick perusal.
>
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893)
> caused 2.7.3 (along with every other Hadoop release) to slip by quite a
> bit. This release's related discussion thread is linked below: [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/
> hdfs-dev%40hadoop.apache.org/index.html#26106 <
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106
> >
> [2]: 2.7.3 release plan: https://www.mail-archive.com/
> hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/
> 6yv2fyrs4jlepmmr>