You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by sryza <gi...@git.apache.org> on 2014/03/08 06:00:10 UTC
[GitHub] spark pull request: SPARK-1064
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/102
SPARK-1064
This reopens PR 649 from incubator-spark against the new repo
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sryza/spark sandy-spark-1064
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/102.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #102
----
commit 552fc04009e15547b315ff8eabbec5c4b1659002
Author: Sandy Ryza <sa...@cloudera.com>
Date: 2014-02-19T00:30:06Z
SPARK-1064. Make it possible to run on YARN without bundling Hadoop jars in Spark assembly
commit 4380ad5b24096f4977bd2d97ff3fde808da4660f
Author: Sandy Ryza <sa...@cloudera.com>
Date: 2014-03-08T04:58:14Z
sbt change
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10506184
--- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala ---
@@ -379,9 +381,48 @@ object ClientBase {
// Based on code from org.apache.hadoop.mapreduce.v2.util.MRApps
def populateHadoopClasspath(conf: Configuration, env: HashMap[String, String]) {
- for (c <- conf.getStrings(YarnConfiguration.YARN_APPLICATION_CLASSPATH)) {
+ val classpathEntries = Option(conf.getStrings(
+ YarnConfiguration.YARN_APPLICATION_CLASSPATH)).getOrElse(
+ getDefaultYarnApplicationClasspath())
+ for (c <- classpathEntries) {
Apps.addToEnvironment(env, Environment.CLASSPATH.name, c.trim)
}
+
+ val mrClasspathEntries = Option(conf.getStrings(
+ "mapreduce.application.classpath")).getOrElse(
+ getDefaultMRApplicationClasspath())
+ if (mrClasspathEntries != null) {
+ for (c <- mrClasspathEntries) {
+ Apps.addToEnvironment(env, Environment.CLASSPATH.name, c.trim)
+ }
+ }
+ }
+
+ def getDefaultYarnApplicationClasspath(): Array[String] = {
+ try {
+ val field = classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH")
--- End diff --
Ah noted, sorry I missed the comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10459998
--- Diff: project/SparkBuild.scala ---
@@ -236,7 +236,8 @@ object SparkBuild extends Build {
"com.novocode" % "junit-interface" % "0.10" % "test",
"org.easymock" % "easymock" % "3.1" % "test",
"org.mockito" % "mockito-all" % "1.8.5" % "test",
- "commons-io" % "commons-io" % "2.4" % "test"
+ "commons-io" % "commons-io" % "2.4" % "test",
+ "org.apache.hadoop" % hadoopClient % hadoopVersion % hadoopScope excludeAll(excludeNetty, excludeAsm, excludeCommonsLogging, excludeSLF4J, excludeOldAsm)
--- End diff --
Yeah, only when HADOOP_PROVIDED is set to true.
Based on a grep for org.apache.hadoop, it's used in core, streaming, yarn, examples, and repl.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37156373
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10460757
--- Diff: project/SparkBuild.scala ---
@@ -236,7 +236,8 @@ object SparkBuild extends Build {
"com.novocode" % "junit-interface" % "0.10" % "test",
"org.easymock" % "easymock" % "3.1" % "test",
"org.mockito" % "mockito-all" % "1.8.5" % "test",
- "commons-io" % "commons-io" % "2.4" % "test"
+ "commons-io" % "commons-io" % "2.4" % "test",
+ "org.apache.hadoop" % hadoopClient % hadoopVersion % hadoopScope excludeAll(excludeNetty, excludeAsm, excludeCommonsLogging, excludeSLF4J, excludeOldAsm)
--- End diff --
When I try adding Hadoop to only those ones, I get a bunch of errors like these:
[error] /home/sandy/spark/spark/bagel/src/main/scala/org/apache/spark/bagel/Bagel.scala:197: bad symbolic reference. A signature in SparkContext.class refers to term io
[error] in package org.apache.hadoop which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling SparkContext.class.
[error] sc, vertices, messages, new DefaultCombiner(), None, part, numPartitions, storageLevel)(
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/102
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10457935
--- Diff: pom.xml ---
@@ -800,5 +800,40 @@
</modules>
</profile>
+
+ <profile>
--- End diff --
A one liner comment here might be nice:
<!-- Build without Hadoop dependencies that are include in some runtime environments. -->
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37374693
Unfortunately, the alpha/stable distinction doesn't fully capture the differences here because the APIs are different between the 0.23 Hadoop line and the 2.0 line, both of which fall under yarn-alpha. The comment above getMapReduceApplicationClasspath explains the differences between 0.23, 2.0, and 2.2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by gzm55 <gi...@git.apache.org>.
Github user gzm55 commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10462859
--- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala ---
@@ -379,7 +381,17 @@ object ClientBase {
// Based on code from org.apache.hadoop.mapreduce.v2.util.MRApps
def populateHadoopClasspath(conf: Configuration, env: HashMap[String, String]) {
- for (c <- conf.getStrings(YarnConfiguration.YARN_APPLICATION_CLASSPATH)) {
+ val classpathEntries = Option(conf.getStrings(
+ YarnConfiguration.YARN_APPLICATION_CLASSPATH)).getOrElse(
+ YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH)
+ for (c <- classpathEntries) {
+ Apps.addToEnvironment(env, Environment.CLASSPATH.name, c.trim)
+ }
+
+ val mrClasspathEntries = Option(conf.getStrings(
+ MRJobConfig.MAPREDUCE_APPLICATION_CLASSPATH)).getOrElse(
+ StringUtils.getStrings(MRJobConfig.DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH))
--- End diff --
Hi, do you mean "getOrElse(MRJobConfig.DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH)" ? I have a compilation error here, saying that StringUtils.getStrings requires a String but gets a Array[String]. And in hadoop CDH 4.6, we have
public static final String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH = {
"$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*",
"$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*",
};
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37090694
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37377235
Looks good thanks @sryza and to @gzm55 for pointing out the issue. @sryza y u keep changing API's?
I'll merge this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10458903
--- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala ---
@@ -379,7 +381,17 @@ object ClientBase {
// Based on code from org.apache.hadoop.mapreduce.v2.util.MRApps
def populateHadoopClasspath(conf: Configuration, env: HashMap[String, String]) {
- for (c <- conf.getStrings(YarnConfiguration.YARN_APPLICATION_CLASSPATH)) {
+ val classpathEntries = Option(conf.getStrings(
--- End diff --
It points to the location on machines in the cluster of the HDFS/YARN/MR code. An admin might add a library like LZO to this, but users should instead be using the distributed cache if there's a jar specific to their application that they want.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10505297
--- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala ---
@@ -379,9 +381,48 @@ object ClientBase {
// Based on code from org.apache.hadoop.mapreduce.v2.util.MRApps
def populateHadoopClasspath(conf: Configuration, env: HashMap[String, String]) {
- for (c <- conf.getStrings(YarnConfiguration.YARN_APPLICATION_CLASSPATH)) {
+ val classpathEntries = Option(conf.getStrings(
+ YarnConfiguration.YARN_APPLICATION_CLASSPATH)).getOrElse(
+ getDefaultYarnApplicationClasspath())
+ for (c <- classpathEntries) {
Apps.addToEnvironment(env, Environment.CLASSPATH.name, c.trim)
}
+
+ val mrClasspathEntries = Option(conf.getStrings(
+ "mapreduce.application.classpath")).getOrElse(
+ getDefaultMRApplicationClasspath())
+ if (mrClasspathEntries != null) {
+ for (c <- mrClasspathEntries) {
+ Apps.addToEnvironment(env, Environment.CLASSPATH.name, c.trim)
+ }
+ }
+ }
+
+ def getDefaultYarnApplicationClasspath(): Array[String] = {
+ try {
+ val field = classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH")
--- End diff --
Unfortunately, the alpha/stable distinction doesn't fully capture the differences here because the APIs are different between the 0.23 Hadoop line and the 2.0 line, both of which fall under yarn-alpha. The comment above getMapReduceApplicationClasspath explains the differences between 0.23, 2.0, and 2.2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37156375
One or more automated tests failed
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13086/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37374812
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37371001
Updated patch takes out the SBT stuff, adds the comment requested by @pwendell, and uses reflection to work around the incompatibilities pointed out by @gzm55
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10457908
--- Diff: project/SparkBuild.scala ---
@@ -236,7 +236,8 @@ object SparkBuild extends Build {
"com.novocode" % "junit-interface" % "0.10" % "test",
"org.easymock" % "easymock" % "3.1" % "test",
"org.mockito" % "mockito-all" % "1.8.5" % "test",
- "commons-io" % "commons-io" % "2.4" % "test"
+ "commons-io" % "commons-io" % "2.4" % "test",
+ "org.apache.hadoop" % hadoopClient % hadoopVersion % hadoopScope excludeAll(excludeNetty, excludeAsm, excludeCommonsLogging, excludeSLF4J, excludeOldAsm)
--- End diff --
Why did this get moved here from `coreSettings`? In the past we've only had the core artifact depend on `hadoop-client` and other things get it via Spark core. This is also how it works in the maven build.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10505246
--- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala ---
@@ -379,9 +381,48 @@ object ClientBase {
// Based on code from org.apache.hadoop.mapreduce.v2.util.MRApps
def populateHadoopClasspath(conf: Configuration, env: HashMap[String, String]) {
- for (c <- conf.getStrings(YarnConfiguration.YARN_APPLICATION_CLASSPATH)) {
+ val classpathEntries = Option(conf.getStrings(
+ YarnConfiguration.YARN_APPLICATION_CLASSPATH)).getOrElse(
+ getDefaultYarnApplicationClasspath())
+ for (c <- classpathEntries) {
Apps.addToEnvironment(env, Environment.CLASSPATH.name, c.trim)
}
+
+ val mrClasspathEntries = Option(conf.getStrings(
+ "mapreduce.application.classpath")).getOrElse(
+ getDefaultMRApplicationClasspath())
+ if (mrClasspathEntries != null) {
+ for (c <- mrClasspathEntries) {
+ Apps.addToEnvironment(env, Environment.CLASSPATH.name, c.trim)
+ }
+ }
+ }
+
+ def getDefaultYarnApplicationClasspath(): Array[String] = {
+ try {
+ val field = classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH")
--- End diff --
Hey @sryza rather than use reflection here why not just modify the `Client.scala` classes in `common` and `stable`? I think the main reason of separating those two was so that we don't have to use reflection everywhere. Is that hard to do in this case?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37089813
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37090695
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13064/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10498244
--- Diff: project/SparkBuild.scala ---
@@ -236,7 +236,8 @@ object SparkBuild extends Build {
"com.novocode" % "junit-interface" % "0.10" % "test",
"org.easymock" % "easymock" % "3.1" % "test",
"org.mockito" % "mockito-all" % "1.8.5" % "test",
- "commons-io" % "commons-io" % "2.4" % "test"
+ "commons-io" % "commons-io" % "2.4" % "test",
+ "org.apache.hadoop" % hadoopClient % hadoopVersion % hadoopScope excludeAll(excludeNetty, excludeAsm, excludeCommonsLogging, excludeSLF4J, excludeOldAsm)
--- End diff --
Let's just drop this in the sbt build for now. We have some other packaging specific code in maven only for bigtop, I think it would fall under that category
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37372654
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10458859
--- Diff: project/SparkBuild.scala ---
@@ -236,7 +236,8 @@ object SparkBuild extends Build {
"com.novocode" % "junit-interface" % "0.10" % "test",
"org.easymock" % "easymock" % "3.1" % "test",
"org.mockito" % "mockito-all" % "1.8.5" % "test",
- "commons-io" % "commons-io" % "2.4" % "test"
+ "commons-io" % "commons-io" % "2.4" % "test",
+ "org.apache.hadoop" % hadoopClient % hadoopVersion % hadoopScope excludeAll(excludeNetty, excludeAsm, excludeCommonsLogging, excludeSLF4J, excludeOldAsm)
--- End diff --
I tried placing it in core settings, but it caused the other modules to fail from not being able to find Hadoop. It must be that core isn't providing it to them in that case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37154500
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37154499
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10459709
--- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala ---
@@ -379,7 +381,17 @@ object ClientBase {
// Based on code from org.apache.hadoop.mapreduce.v2.util.MRApps
def populateHadoopClasspath(conf: Configuration, env: HashMap[String, String]) {
- for (c <- conf.getStrings(YarnConfiguration.YARN_APPLICATION_CLASSPATH)) {
+ val classpathEntries = Option(conf.getStrings(
--- End diff --
Okay sounds good. Just wanted to make sure this wasn't the main path for user application code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10478785
--- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala ---
@@ -379,7 +381,17 @@ object ClientBase {
// Based on code from org.apache.hadoop.mapreduce.v2.util.MRApps
def populateHadoopClasspath(conf: Configuration, env: HashMap[String, String]) {
- for (c <- conf.getStrings(YarnConfiguration.YARN_APPLICATION_CLASSPATH)) {
+ val classpathEntries = Option(conf.getStrings(
+ YarnConfiguration.YARN_APPLICATION_CLASSPATH)).getOrElse(
+ YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH)
+ for (c <- classpathEntries) {
+ Apps.addToEnvironment(env, Environment.CLASSPATH.name, c.trim)
+ }
+
+ val mrClasspathEntries = Option(conf.getStrings(
+ MRJobConfig.MAPREDUCE_APPLICATION_CLASSPATH)).getOrElse(
+ StringUtils.getStrings(MRJobConfig.DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH))
--- End diff --
Oy, good point. In the 2.0 line (which CDH4 is based off of) it's an array of Strings, and in 2.2/GA is's a String. And in the 0.23 line, it doesn't exist at all.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37374814
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13121/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37372653
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37089812
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/102#issuecomment-37154606
Updated version works with sbt as well
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10459510
--- Diff: project/SparkBuild.scala ---
@@ -236,7 +236,8 @@ object SparkBuild extends Build {
"com.novocode" % "junit-interface" % "0.10" % "test",
"org.easymock" % "easymock" % "3.1" % "test",
"org.mockito" % "mockito-all" % "1.8.5" % "test",
- "commons-io" % "commons-io" % "2.4" % "test"
+ "commons-io" % "commons-io" % "2.4" % "test",
+ "org.apache.hadoop" % hadoopClient % hadoopVersion % hadoopScope excludeAll(excludeNetty, excludeAsm, excludeCommonsLogging, excludeSLF4J, excludeOldAsm)
--- End diff --
Does the failure only happen when using your "provided" stuff? If so, the issue is that the sub projects won't inherit those anymore because I don't think provided dependencies are inherited in sbt.
It might be nice to scope this more narrowly then and only do it for the modules which actually depend on hadoop. Is it a lot of them?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1064
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/102#discussion_r10458176
--- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala ---
@@ -379,7 +381,17 @@ object ClientBase {
// Based on code from org.apache.hadoop.mapreduce.v2.util.MRApps
def populateHadoopClasspath(conf: Configuration, env: HashMap[String, String]) {
- for (c <- conf.getStrings(YarnConfiguration.YARN_APPLICATION_CLASSPATH)) {
+ val classpathEntries = Option(conf.getStrings(
--- End diff --
Hey just so I understand here. What is the way that YARN_APPLICATION_CLASSPATH and MAPREDUCE_APPLICATION_CCLASSPATH are used? Is this just designed to point to the locally installed Yarn/MR code? Or do users every go and include their own application code at these locations as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---