You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gweidner <gi...@git.apache.org> on 2015/10/16 02:52:18 UTC
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
GitHub user gweidner opened a pull request:
https://github.com/apache/spark/pull/9144
[SPARK-11109] [CORE] Move FsHistoryProvider off deprecated AccessControlException
Switched from deprecated org.apache.hadoop.fs.permission.AccessControlException to org.apache.hadoop.security.AccessControlException.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gweidner/spark SPARK-11109
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9144.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9144
----
commit 2f2754142df3feefacc7e91f946cdc1876843e3d
Author: gweidner <gw...@us.ibm.com>
Date: 2015-10-16T00:17:02Z
[SPARK-11109] [CORE] Move FsHistoryProvider off
org.apache.hadoop.fs.permission.AccessControlException
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9144
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9144#issuecomment-148563763
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/9144#issuecomment-148563344
Does this class exist in older versions of Hadoop, like 1.1 etc?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9144#issuecomment-148791764
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/9144#issuecomment-148662454
It's been in 1.x. In fact, it's been in 0.19 when the {{fs.permission}} one was first tagged as deprecated [fs/permission/AccessControlException.java](https://github.com/apache/hadoop/blob/branch-0.19/src/core/org/apache/hadoop/fs/permission/AccessControlException.java)
Hadoop 3.x finally cut the class (more precisely, preserved its non-deprecated its child and rm'd the ancestor); it was a build against 3.0 which showed this {{mvn -Pyarn,hadoop-2.6 -Dhadoop.version=3.0.0-SNAPSHOT}}... that's the way to verify things are fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9144#issuecomment-148835677
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43843/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9144#issuecomment-148797797
[Test build #43843 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43843/consoleFull) for PR 9144 at commit [`2f27541`](https://github.com/apache/spark/commit/2f2754142df3feefacc7e91f946cdc1876843e3d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9144#issuecomment-148790293
LGTM. ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9144#issuecomment-148835387
[Test build #43843 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43843/console) for PR 9144 at commit [`2f27541`](https://github.com/apache/spark/commit/2f2754142df3feefacc7e91f946cdc1876843e3d).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9144#issuecomment-148835676
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by gweidner <gi...@git.apache.org>.
Github user gweidner commented on the pull request:
https://github.com/apache/spark/pull/9144#issuecomment-148568516
Thank you Hari for the quick input. Yes - I downloaded/extracted hadoop-1.0.0.tar.gz dated 2011-12-15 from https://archive.apache.org/dist/hadoop/common/hadoop-1.0.0/ and verified that org.apache.hadoop.security.AccessControlException was present.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9144#issuecomment-148791740
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9144#issuecomment-148836167
Merging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/9144#issuecomment-148789432
@vanzin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by gweidner <gi...@git.apache.org>.
Github user gweidner commented on the pull request:
https://github.com/apache/spark/pull/9144#issuecomment-148701565
Thank you Steve for the detailed hadoop information. For the below build command on my local environment:
./build/mvn -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=3.0.0-SNAPSHOT -DskipTests clean package
**without** this change the following error was observed:
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-core_2.10 ---
[INFO] Using zinc server for incremental compilation
[info] Compiling 484 Scala sources and 63 Java sources to /home/bigdata/clone/spark/core/target/scala-2.10/classes...
[error] /home/bigdata/clone/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:31: object AccessControlException is not a member of package org.apache.hadoop.fs.permission
[error] import org.apache.hadoop.fs.permission.AccessControlException
[error] ^
[error] /home/bigdata/clone/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:193: not found: type AccessControlException
[error] case e: AccessControlException =>
[error] ^
[error] /home/bigdata/clone/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:416: not found: type AccessControlException
[error] case e: AccessControlException =>
[error] ^
[error] three errors found
[error] Compile failed at Oct 16, 2015 3:42:38 AM [16.688s]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 3.358 s]
[INFO] Spark Project Test Tags ............................ SUCCESS [ 2.324 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 8.606 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 8.760 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 5.803 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 7.034 s]
[INFO] Spark Project Core ................................. FAILURE [ 19.885 s]
[INFO] Spark Project Bagel ................................ SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project YARN Shuffle Service ................. SKIPPED
[INFO] Spark Project YARN ................................. SKIPPED
[INFO] Spark Project Hive Thrift Server ................... SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Project External Twitter ..................... SKIPPED
[INFO] Spark Project External Flume Sink .................. SKIPPED
[INFO] Spark Project External Flume ....................... SKIPPED
[INFO] Spark Project External Flume Assembly .............. SKIPPED
[INFO] Spark Project External MQTT ........................ SKIPPED
[INFO] Spark Project External MQTT Assembly ............... SKIPPED
[INFO] Spark Project External ZeroMQ ...................... SKIPPED
[INFO] Spark Project External Kafka ....................... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Project External Kafka Assembly .............. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 56.924 s
[INFO] Finished at: 2015-10-16T03:42:38-08:00
[INFO] Final Memory: 70M/716M
[INFO] ------------------------------------------------------------------------
**with** this change, the same build command with -Dhadoop.version=3.0.0-SNAPSHOT was successful:
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 3.385 s]
[INFO] Spark Project Test Tags ............................ SUCCESS [ 2.349 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 8.311 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 8.016 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 4.997 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 6.269 s]
[INFO] Spark Project Core ................................. SUCCESS [02:06 min]
[INFO] Spark Project Bagel ................................ SUCCESS [ 4.202 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 13.597 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 33.571 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [ 51.255 s]
[INFO] Spark Project SQL .................................. SUCCESS [01:04 min]
[INFO] Spark Project ML Library ........................... SUCCESS [01:13 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 2.385 s]
[INFO] Spark Project Hive ................................. SUCCESS [ 47.667 s]
[INFO] Spark Project REPL ................................. SUCCESS [ 8.235 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 8.308 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 10.561 s]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [ 7.959 s]
[INFO] Spark Project Assembly ............................. SUCCESS [02:05 min]
[INFO] Spark Project External Twitter ..................... SUCCESS [ 7.346 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [ 4.635 s]
[INFO] Spark Project External Flume ....................... SUCCESS [ 8.602 s]
[INFO] Spark Project External Flume Assembly .............. SUCCESS [ 2.853 s]
[INFO] Spark Project External MQTT ........................ SUCCESS [ 16.184 s]
[INFO] Spark Project External MQTT Assembly ............... SUCCESS [ 6.149 s]
[INFO] Spark Project External ZeroMQ ...................... SUCCESS [ 6.536 s]
[INFO] Spark Project External Kafka ....................... SUCCESS [ 10.617 s]
[INFO] Spark Project Examples ............................. SUCCESS [01:47 min]
[INFO] Spark Project External Kafka Assembly .............. SUCCESS [ 7.777 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:10 min
[INFO] Finished at: 2015-10-16T04:01:22-08:00
[INFO] Final Memory: 414M/1703M
[INFO] ------------------------------------------------------------------------
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11109] [CORE] Move FsHistoryProvider of...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/9144#issuecomment-148717590
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org