You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by lianhuiwang <gi...@git.apache.org> on 2015/01/09 14:14:19 UTC

[GitHub] spark pull request: [SPARK-5173]support python application running...

GitHub user lianhuiwang opened a pull request:

    https://github.com/apache/spark/pull/3976

    [SPARK-5173]support python application running on yarn cluster mode

    now when we run python application on yarn cluster mode through spark-submit, spark-submit does not support python application on yarn cluster mode. so i modify code of submit and yarn's AM in order to support it.@JoshRosen @tdas 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/lianhuiwang/spark SPARK-5173

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/3976.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3976
    
----
commit 9c941bc59527e594ee1d155c00cb8e55d7c40fe8
Author: lianhuiwang <li...@gmail.com>
Date:   2015-01-09T12:58:24Z

    support python application running on yarn cluster mode

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-69355774
  
      [Test build #25326 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25326/consoleFull) for   PR 3976 at commit [`172eec1`](https://github.com/apache/spark/commit/172eec10b9daaf9ed838e821474d28871ab63462).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72240509
  
    This looks like the right approach.  Added some comments inline.  Are you able to add a test for this in `YarnClusterSuite`?
    
    Also, one last small thing: in `PythonRunner` are you able to remove the reference to "Spark submit" in the header comment, as this is now used in a more general way? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23904826
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -134,12 +136,29 @@ object SparkSubmit {
           }
         }
     
    +    val isYarnCluster = clusterManager == YARN && deployMode == CLUSTER
    +
    +    // Require all python files to be local, so we can add them to the PYTHONPATH
    +    // when yarn-cluster, all python files can be non-local
    +    if (args.isPython && !isYarnCluster) {
    +      if (Utils.nonLocalPaths(args.primaryResource).nonEmpty) {
    +        SparkSubmit.printErrorAndExit(
    +          s"Only local python files are supported: $args.primaryResource")
    +      }
    +      val nonLocalPyFiles = Utils.nonLocalPaths(args.pyFiles).mkString(",")
    +      if (nonLocalPyFiles.nonEmpty) {
    +        SparkSubmit.printErrorAndExit(
    +          s"Only local additional python files are supported: $nonLocalPyFiles")
    +      }
    +    }
    +
         // The following modes are not supported or applicable
         (clusterManager, deployMode) match {
           case (MESOS, CLUSTER) =>
             printErrorAndExit("Cluster deploy mode is currently not supported for Mesos clusters.")
    -      case (_, CLUSTER) if args.isPython =>
    -        printErrorAndExit("Cluster deploy mode is currently not supported for python applications.")
    +      case (STANDALONE, CLUSTER) if args.isPython =>
    +        printErrorAndExit("Cluster deploy mode is currently not supported for python " +
    +          "applications on Standalone clusters")
    --- End diff --
    
    also, I think it's ok to use lower case for `standalone clusters`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23857834
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -148,7 +151,7 @@ object SparkSubmit {
         }
     
         // If we're running a python app, set the main class to our specific python runner
    -    if (args.isPython) {
    +    if (args.isPython && deployMode != CLUSTER) {
    --- End diff --
    
    nit: use "deployMode == CLIENT"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-70770431
  
    yes, we just specify .py file or primaryResource file via spark-submit, with this PR we can make pyspark run in yarn-cluster mode.
    example:
    spark-submit --master yarn-master --num-executors 1 --driver-memory 1g --executor-memory 1g  xx.py --primaryResource yy.conf
    this config should not be changed and is same as pyspark on yarn-client mode.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72302620
  
    @sryza @andrewor14  thank for your reviews. I have updated with your comments. Later i will add YarnClusterSuite for a test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23861241
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -138,8 +140,9 @@ object SparkSubmit {
         (clusterManager, deployMode) match {
           case (MESOS, CLUSTER) =>
             printErrorAndExit("Cluster deploy mode is currently not supported for Mesos clusters.")
    -      case (_, CLUSTER) if args.isPython =>
    -        printErrorAndExit("Cluster deploy mode is currently not supported for python applications.")
    +      case (STANDALONE, CLUSTER) if args.isPython =>
    +        printErrorAndExit("Standalone-Cluster deploy mode is currently not supported" +
    --- End diff --
    
    yes please try to be consistent with other error messages here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23856503
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -138,8 +140,9 @@ object SparkSubmit {
         (clusterManager, deployMode) match {
           case (MESOS, CLUSTER) =>
             printErrorAndExit("Cluster deploy mode is currently not supported for Mesos clusters.")
    -      case (_, CLUSTER) if args.isPython =>
    -        printErrorAndExit("Cluster deploy mode is currently not supported for python applications.")
    +      case (STANDALONE, CLUSTER) if args.isPython =>
    +        printErrorAndExit("Standalone-Cluster deploy mode is currently not supported" +
    --- End diff --
    
    Nit: to mirror the above messages a little better, can this be "Cluster deploy mode is currently not supported for python applications on Standalone clusters"?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23904387
  
    --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
    @@ -430,6 +430,10 @@ private[spark] class ApplicationMaster(args: ApplicationMasterArguments,
       private def startUserClass(): Thread = {
         logInfo("Starting the user JAR in a separate Thread")
    --- End diff --
    
    no, what I mean is it will become outdated with your changes, since we won't be "starting the user JAR" if we're running a python application


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72315747
  
      [Test build #26462 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26462/consoleFull) for   PR 3976 at commit [`5b30064`](https://github.com/apache/spark/commit/5b300648fe53d9de604e8afce7580fddfe6bbaef).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72349753
  
      [Test build #26470 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26470/consoleFull) for   PR 3976 at commit [`47d2fc3`](https://github.com/apache/spark/commit/47d2fc35e53a8851790607085bc67e94736358d6).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23865116
  
    --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
    @@ -430,6 +430,10 @@ private[spark] class ApplicationMaster(args: ApplicationMasterArguments,
       private def startUserClass(): Thread = {
         logInfo("Starting the user JAR in a separate Thread")
    --- End diff --
    
    this comment is outdated


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-69332770
  
      [Test build #25325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25325/consoleFull) for   PR 3976 at commit [`9c941bc`](https://github.com/apache/spark/commit/9c941bc59527e594ee1d155c00cb8e55d7c40fe8).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23857876
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -267,10 +277,22 @@ object SparkSubmit {
         // In yarn-cluster mode, use yarn.Client as a wrapper around the user class
         if (isYarnCluster) {
           childMainClass = "org.apache.spark.deploy.yarn.Client"
    -      if (args.primaryResource != SPARK_INTERNAL) {
    -        childArgs += ("--jar", args.primaryResource)
    +      if (args.isPython) {// yarn-cluster mode for python application
    --- End diff --
    
    Take this comment out


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23904930
  
    --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala ---
    @@ -81,6 +91,9 @@ class ApplicationMasterArguments(val args: Array[String]) {
           |Options:
           |  --jar JAR_PATH       Path to your application's JAR file
           |  --class CLASS_NAME   Name of your application's main class
    +      |  --primary-py-file    A primary Python file
    --- End diff --
    
    The main python file


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72398202
  
      [Test build #26480 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26480/consoleFull) for   PR 3976 at commit [`2385ef6`](https://github.com/apache/spark/commit/2385ef679638fcb0b544a3de7744c9f4f2c242f0).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23856550
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -165,6 +168,13 @@ object SparkSubmit {
           }
         }
     
    +    // In yarn-cluster mode for a python app, add primary Resource and pyFiles to files
    --- End diff --
    
    nit: don't capitalize "Resource"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72349756
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26470/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23904494
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -267,10 +292,20 @@ object SparkSubmit {
         // In yarn-cluster mode, use yarn.Client as a wrapper around the user class
         if (isYarnCluster) {
           childMainClass = "org.apache.spark.deploy.yarn.Client"
    -      if (args.primaryResource != SPARK_INTERNAL) {
    -        childArgs += ("--jar", args.primaryResource)
    +      if (args.isPython) {
    +        val mainPyFile = new Path(args.primaryResource).getName
    +        childArgs += ("--primary-py-file", mainPyFile)
    +        if (args.pyFiles != null) {
    +          val pyFilesNames = args.pyFiles.split(",").map(p => (new Path(p)).getName).mkString(",")
    +          childArgs += ("--py-files", pyFilesNames)
    +        }
    +        childArgs += ("--class", "org.apache.spark.deploy.PythonRunner")
    --- End diff --
    
    why i do not use classOf here. because PythonRunner is a object, it can be used on classOf. @sryza 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23904909
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -134,12 +136,29 @@ object SparkSubmit {
           }
         }
     
    +    val isYarnCluster = clusterManager == YARN && deployMode == CLUSTER
    +
    +    // Require all python files to be local, so we can add them to the PYTHONPATH
    +    // when yarn-cluster, all python files can be non-local
    --- End diff --
    
    ```
    // In YARN cluster mode, python files are distributed as regular files, which can be non-local
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72531678
  
    Alright, LGTM thanks a lot @lianhuiwang I'm merging this into master. We can fix the SPARK_HOME issue separately.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72532026
  
    Ok LGTM thanks a lot @lianhuiwang I'm merging this into master. We can fix the SPARK_HOME issue separately later. Could you file a JIRA for it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23904936
  
    --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala ---
    @@ -185,6 +192,7 @@ private[spark] class ClientArguments(args: Array[String], sparkConf: SparkConf)
           |  --jar JAR_PATH           Path to your application's JAR file (required in yarn-cluster
           |                           mode)
           |  --class CLASS_NAME       Name of your application's main class (required)
    +      |  --primary-py-file        A primary Python file
    --- End diff --
    
    same here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72302649
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26452/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23904843
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -134,12 +136,29 @@ object SparkSubmit {
           }
         }
     
    +    val isYarnCluster = clusterManager == YARN && deployMode == CLUSTER
    +
    +    // Require all python files to be local, so we can add them to the PYTHONPATH
    +    // when yarn-cluster, all python files can be non-local
    +    if (args.isPython && !isYarnCluster) {
    +      if (Utils.nonLocalPaths(args.primaryResource).nonEmpty) {
    +        SparkSubmit.printErrorAndExit(
    +          s"Only local python files are supported: $args.primaryResource")
    +      }
    +      val nonLocalPyFiles = Utils.nonLocalPaths(args.pyFiles).mkString(",")
    +      if (nonLocalPyFiles.nonEmpty) {
    +        SparkSubmit.printErrorAndExit(
    +          s"Only local additional python files are supported: $nonLocalPyFiles")
    +      }
    +    }
    +
         // The following modes are not supported or applicable
         (clusterManager, deployMode) match {
           case (MESOS, CLUSTER) =>
             printErrorAndExit("Cluster deploy mode is currently not supported for Mesos clusters.")
    -      case (_, CLUSTER) if args.isPython =>
    -        printErrorAndExit("Cluster deploy mode is currently not supported for python applications.")
    +      case (STANDALONE, CLUSTER) if args.isPython =>
    +        printErrorAndExit("Cluster deploy mode is currently not supported for python " +
    +          "applications on Standalone clusters")
    --- End diff --
    
    thanks. @andrewor14 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23864907
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -267,10 +277,22 @@ object SparkSubmit {
         // In yarn-cluster mode, use yarn.Client as a wrapper around the user class
         if (isYarnCluster) {
           childMainClass = "org.apache.spark.deploy.yarn.Client"
    -      if (args.primaryResource != SPARK_INTERNAL) {
    -        childArgs += ("--jar", args.primaryResource)
    +      if (args.isPython) {// yarn-cluster mode for python application
    +      val primaryResourceLocalPath = new Path(args.primaryResource)
    +        childArgs += ("--primaryResource", primaryResourceLocalPath.getName)
    +        val pyFilesLocalNames:String = if (args.pyFiles != null) {
    +          args.pyFiles.split(",").map { p => (new Path(p)).getName }.mkString(",")
    --- End diff --
    
    @lianhuiwang quick question are you stripping the path prefix here because all the python files in YARN are already found in the working directory? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23861918
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
    @@ -172,7 +172,8 @@ private[spark] class SparkSubmitArguments(args: Seq[String], env: Map[String, St
         }
     
         // Require all python files to be local, so we can add them to the PYTHONPATH
    -    if (isPython) {
    +    // when yarn-cluster, all python files can be non-local
    +    if (isPython && !master.equalsIgnoreCase("yarn-cluster")) {
    --- End diff --
    
    you'll probably need to `def isYarnClusterMode` or something and move the relevant code from SparkSubmit here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72402028
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26480/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23862207
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -165,6 +168,13 @@ object SparkSubmit {
           }
         }
     
    +    // In yarn-cluster mode for a python app, add primary Resource and pyFiles to files
    +    // that can be distributed with the job
    +    if (args.isPython && deployMode == CLUSTER) {
    --- End diff --
    
    this check doesn't match the comment above, which says `yarn-cluster` mode


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72437808
  
      [Test build #26508 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26508/consoleFull) for   PR 3976 at commit [`28a8a58`](https://github.com/apache/spark/commit/28a8a587492746bdb7acca0fef975e896cc34200).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72402026
  
      [Test build #26480 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26480/consoleFull) for   PR 3976 at commit [`2385ef6`](https://github.com/apache/spark/commit/2385ef679638fcb0b544a3de7744c9f4f2c242f0).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-70695056
  
    can you describe this in a bit more detail?  Is this intended to make pyspark run in cluster mode or just if you specify .py file via spark-submit.
    
    What type of testing did you do on this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-85498686
  
    > @andrewor14 for python application, if SPARK_HOME of submission node is different from the nodeManager, so it can not work in my test. example:submission node's version is 1.2, but spark's version in nodemanger is 1.1, that is can not work now. i think there is other problem that is not belong to this PR,because in yarn client mode it is also exist.
    
    @lianhuiwang  Sorry for bringing this up after this is closed but does this mean that spark has to be installed on all the nodes?  That shouldn't be needed on YARN.  
    Note that when I try to run an example in cluster mode on yarn I get the following error, is this known?
    
    Traceback (most recent call last):
      File "test.py", line 11, in <module>
        sc = SparkContext(appName="PythonWordCount")
      File "/grid/6/tmp/yarn-local/usercache/tgraves/filecache/26/spark-assembly-1.3.0.0-hadoop2.6.0.6.1502061521.jar/pyspark/context.py", line 108, in __init__
      File "/grid/6/tmp/yarn-local/usercache/tgraves/filecache/26/spark-assembly-1.3.0.0-hadoop2.6.0.6.1502061521.jar/pyspark/context.py", line 222, in _ensure_initialized
      File "/grid/6/tmp/yarn-local/usercache/tgraves/filecache/26/spark-assembly-1.3.0.0-hadoop2.6.0.6.1502061521.jar/pyspark/java_gateway.py", line 32, in launch_gateway
      File "/usr/lib64/python2.6/UserDict.py", line 22, in __getitem__
        raise KeyError(key)
    KeyError: 'SPARK_HOME'
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72304811
  
      [Test build #26453 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26453/consoleFull) for   PR 3976 at commit [`097a5ec`](https://github.com/apache/spark/commit/097a5ec37456bf9d13a952f4108a750b9f9f84d0).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72304815
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26453/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23857534
  
    --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala ---
    @@ -103,11 +104,15 @@ private[spark] class ClientArguments(args: Array[String], sparkConf: SparkConf)
               userClass = value
               args = tail
     
    +        case ("--primaryResource") :: value :: tail =>
    --- End diff --
    
    Looked deeper and realized this isn't a spark-submit argument and that it's just used for the YARN stuff internally.
    
    primaryResource is kind of a confusing name because a userJar is also a primary resource.  Maybe "main-py-module" instead? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23862118
  
    --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala ---
    @@ -103,11 +104,15 @@ private[spark] class ClientArguments(args: Array[String], sparkConf: SparkConf)
               userClass = value
               args = tail
     
    +        case ("--primaryResource") :: value :: tail =>
    --- End diff --
    
    I agree, since we don't ever set this to the main jar if we're not running python


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-70851442
  
    Sorry its not clear to me, by pyspark I meant the interactive /bin/pyspark, is this intended to make the interactive python shell work or just batch mode specify the python file?  By the looks of it I'm assuming just the batch mode.  



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23884756
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -267,10 +277,22 @@ object SparkSubmit {
         // In yarn-cluster mode, use yarn.Client as a wrapper around the user class
         if (isYarnCluster) {
           childMainClass = "org.apache.spark.deploy.yarn.Client"
    -      if (args.primaryResource != SPARK_INTERNAL) {
    -        childArgs += ("--jar", args.primaryResource)
    +      if (args.isPython) {// yarn-cluster mode for python application
    +      val primaryResourceLocalPath = new Path(args.primaryResource)
    +        childArgs += ("--primaryResource", primaryResourceLocalPath.getName)
    +        val pyFilesLocalNames:String = if (args.pyFiles != null) {
    +          args.pyFiles.split(",").map { p => (new Path(p)).getName }.mkString(",")
    --- End diff --
    
    @andrewor14 yes, i strip the path prefix, because files will download on working directory when yarn cluster mode.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72415530
  
      [Test build #26490 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26490/consoleFull) for   PR 3976 at commit [`67f8cee`](https://github.com/apache/spark/commit/67f8cee9e25b5bd05c0252705b1f67cb63b0fa01).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72428343
  
      [Test build #26508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26508/consoleFull) for   PR 3976 at commit [`28a8a58`](https://github.com/apache/spark/commit/28a8a587492746bdb7acca0fef975e896cc34200).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72374353
  
      [Test build #26474 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26474/consoleFull) for   PR 3976 at commit [`03640ab`](https://github.com/apache/spark/commit/03640ab5c9c9b6890b4b68cf7aaf2183444bd6ed).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-85542318
  
    @tgravescs  thanks, i know what you means. i will take a look at it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72327242
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26466/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72016555
  
      [Test build #26313 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26313/consoleFull) for   PR 3976 at commit [`f1f55b6`](https://github.com/apache/spark/commit/f1f55b6eb4b65499be8e182e857d89a158873234).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72321243
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26464/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23856668
  
    --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala ---
    @@ -103,11 +104,15 @@ private[spark] class ClientArguments(args: Array[String], sparkConf: SparkConf)
               userClass = value
               args = tail
     
    +        case ("--primaryResource") :: value :: tail =>
    --- End diff --
    
    What's the need for this new argument?  The argument that's not an option is implicitly understood to be the primary resource.  Is there a reason this won't work for python?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23904929
  
    --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
    @@ -430,6 +430,10 @@ private[spark] class ApplicationMaster(args: ApplicationMasterArguments,
       private def startUserClass(): Thread = {
         logInfo("Starting the user JAR in a separate Thread")
    --- End diff --
    
    ok, i got it. i will update it. thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23905022
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -134,12 +136,29 @@ object SparkSubmit {
           }
         }
     
    +    val isYarnCluster = clusterManager == YARN && deployMode == CLUSTER
    +
    +    // Require all python files to be local, so we can add them to the PYTHONPATH
    +    // when yarn-cluster, all python files can be non-local
    +    if (args.isPython && !isYarnCluster) {
    +      if (Utils.nonLocalPaths(args.primaryResource).nonEmpty) {
    +        SparkSubmit.printErrorAndExit(
    --- End diff --
    
    if we move it to SparkSubmitArguments, we need to get clusterManager and deployMode. But this work has been did in SparkSubmit. there are some repeated works. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by voukka <gi...@git.apache.org>.
Github user voukka commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72173406
  
    @lianhuiwang thank you for non-local files feature.
    
    Could admins review this PR and merge it, please?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72415533
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26490/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72401201
  
    for python application, if SPARK_HOME of submission node is different from the nodeManager, so it can not work in my test. example:submission node's version is 1.2, but spark's version in nodemanger is 1.1, that is can not work now. i think there is other problem that is not belong to this PR,because in yarn client mode it is also exist.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72578648
  
    @andrewor14 thank you, About SPARK_HOME, we need to consider two places of compatible. The first is communication  between python context and Scala context. The second is communication between PythonRDD and Python process. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
GitHub user lianhuiwang reopened a pull request:

    https://github.com/apache/spark/pull/3976

    [SPARK-5173]support python application running on yarn cluster mode

    now when we run python application on yarn cluster mode through spark-submit, spark-submit does not support python application on yarn cluster mode. so i modify code of submit and yarn's AM in order to support it.
    through specifying .py file or primaryResource file via spark-submit, we can make pyspark run in yarn-cluster mode.
    example:spark-submit --master yarn-master --num-executors 1 --driver-memory 1g --executor-memory 1g  xx.py --primaryResource yy.conf
    this config is same as pyspark on yarn-client mode.
    firstly,we put local path of .py or primaryResource to yarn's dist.files.that can be distributed on slave nodes.and then in spark-submit we transfer --py-files and --primaryResource to yarn.Client and use "org.apache.spark.deploy.PythonRunner" to user class that can run .py files on ApplicationMaster.
    in yarn.Client we transfer --py-files and --primaryResource to  ApplicationMaster.
    in ApplicationMaster, user's class is org.apache.spark.deploy.PythonRunner, and user's args is primaryResource and -py-files. so that can make pyspark run on ApplicationMaster.
    @JoshRosen @tgravescs @sryza

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/lianhuiwang/spark SPARK-5173

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/3976.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3976
    
----
commit 9c941bc59527e594ee1d155c00cb8e55d7c40fe8
Author: lianhuiwang <li...@gmail.com>
Date:   2015-01-09T12:58:24Z

    support python application running on yarn cluster mode

commit 172eec10b9daaf9ed838e821474d28871ab63462
Author: Wang Lianhui <li...@gmail.com>
Date:   2015-01-09T15:01:52Z

    fix a min submit's bug

commit f1f55b6eb4b65499be8e182e857d89a158873234
Author: lianhuiwang <li...@gmail.com>
Date:   2015-01-29T11:13:35Z

    when yarn-cluster, all python files can be non-local

commit 905a10610532578c774e58d12b927597330fb9ff
Author: lianhuiwang <li...@gmail.com>
Date:   2015-01-31T03:29:09Z

    update with sryza and andrewor 's comments

commit 097a5ec37456bf9d13a952f4108a750b9f9f84d0
Author: lianhuiwang <li...@gmail.com>
Date:   2015-01-31T03:59:06Z

    fix line length exceeds 100

commit 5b300648fe53d9de604e8afce7580fddfe6bbaef
Author: lianhuiwang <li...@gmail.com>
Date:   2015-01-31T12:18:22Z

    add test

commit d60bc6069cf65637622472ef1cd27153333df53c
Author: lianhuiwang <li...@gmail.com>
Date:   2015-01-31T14:07:03Z

    fix test

commit 2adc8f591ddd0f253496c18d32b1910d29e04c8d
Author: lianhuiwang <li...@gmail.com>
Date:   2015-01-31T16:35:01Z

    add spark.test.home

commit 47d2fc35e53a8851790607085bc67e94736358d6
Author: lianhuiwang <li...@gmail.com>
Date:   2015-02-01T02:40:25Z

    fix test

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23862088
  
    --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala ---
    @@ -48,6 +50,14 @@ class ApplicationMasterArguments(val args: Array[String]) {
               userClass = value
               args = tail
     
    +        case ("--primaryResource") :: value :: tail =>
    --- End diff --
    
    this should be `--primary-resource` instead of camel case for consistency


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72374357
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26474/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23865157
  
    --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
    @@ -430,6 +430,10 @@ private[spark] class ApplicationMaster(args: ApplicationMasterArguments,
       private def startUserClass(): Thread = {
         logInfo("Starting the user JAR in a separate Thread")
    --- End diff --
    
    actually the method name `startUserClass` is now outdated too. It should be `startUserApplication`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23899688
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -267,10 +292,22 @@ object SparkSubmit {
         // In yarn-cluster mode, use yarn.Client as a wrapper around the user class
         if (isYarnCluster) {
           childMainClass = "org.apache.spark.deploy.yarn.Client"
    -      if (args.primaryResource != SPARK_INTERNAL) {
    -        childArgs += ("--jar", args.primaryResource)
    +      if (args.isPython) {
    +        val mainPyFile = new Path(args.primaryResource).getName
    +        childArgs += ("--primary-py-file", mainPyFile)
    +        val pyFilesNames: String = if (args.pyFiles != null) {
    +          args.pyFiles.split(",").map { p => (new Path(p)).getName }.mkString(",")
    +        } else {
    +          "null"
    +        }
    +        childArgs += ("--py-files", pyFilesNames)
    +        childArgs += ("--class", "org.apache.spark.deploy.PythonRunner")
    --- End diff --
    
    nit: use classOf[PythonRunner]


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23899908
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -267,10 +292,22 @@ object SparkSubmit {
         // In yarn-cluster mode, use yarn.Client as a wrapper around the user class
         if (isYarnCluster) {
           childMainClass = "org.apache.spark.deploy.yarn.Client"
    -      if (args.primaryResource != SPARK_INTERNAL) {
    -        childArgs += ("--jar", args.primaryResource)
    +      if (args.isPython) {
    +        val mainPyFile = new Path(args.primaryResource).getName
    +        childArgs += ("--primary-py-file", mainPyFile)
    +        val pyFilesNames: String = if (args.pyFiles != null) {
    --- End diff --
    
    Why not do
    
        if (args.pyFiles != null) {
          val pyFilesNames = args.pyFiles.split...
          childArgs += ("--py-files", pyFilesNames)
        }
    
    instead of setting it to "null"?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23888565
  
    --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
    @@ -430,6 +430,10 @@ private[spark] class ApplicationMaster(args: ApplicationMasterArguments,
       private def startUserClass(): Thread = {
         logInfo("Starting the user JAR in a separate Thread")
    --- End diff --
    
    https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L451
    @andrewor14  here is not outdated. this is same to master's code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-69332862
  
      [Test build #25325 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25325/consoleFull) for   PR 3976 at commit [`9c941bc`](https://github.com/apache/spark/commit/9c941bc59527e594ee1d155c00cb8e55d7c40fe8).
     * This patch **fails Scala style tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72399545
  
      [Test build #26482 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26482/consoleFull) for   PR 3976 at commit [`0319ae3`](https://github.com/apache/spark/commit/0319ae328b2db694684ea586cbb7d49fb2b487c7).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23899955
  
    --- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala ---
    @@ -98,6 +121,9 @@ class YarnClusterSuite extends FunSuite with BeforeAndAfterAll with Matchers wit
         }
     
         fakeSparkJar = File.createTempFile("sparkJar", null, tempDir)
    +    val sparkHome = sys.props.getOrElse("spark.test.home", fail("spark.test.home is not set!"))
    --- End diff --
    
    What's the motivation for these additions?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72302648
  
      [Test build #26452 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26452/consoleFull) for   PR 3976 at commit [`905a106`](https://github.com/apache/spark/commit/905a10610532578c774e58d12b927597330fb9ff).
     * This patch **fails Scala style tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-69345031
  
      [Test build #25326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25326/consoleFull) for   PR 3976 at commit [`172eec1`](https://github.com/apache/spark/commit/172eec10b9daaf9ed838e821474d28871ab63462).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-73541324
  
    @tgravescs your thought is right. but there is just different at Yarn internal's Client. it is same way on spark-submit with Yarn client and Yarn cluster.so i think there is no different on user documentation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72437821
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26508/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-85536800
  
    filed https://issues.apache.org/jira/browse/SPARK-6506 for this


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang closed the pull request at:

    https://github.com/apache/spark/pull/3976


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72008811
  
      [Test build #26313 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26313/consoleFull) for   PR 3976 at commit [`f1f55b6`](https://github.com/apache/spark/commit/f1f55b6eb4b65499be8e182e857d89a158873234).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23904922
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -134,12 +136,29 @@ object SparkSubmit {
           }
         }
     
    +    val isYarnCluster = clusterManager == YARN && deployMode == CLUSTER
    +
    +    // Require all python files to be local, so we can add them to the PYTHONPATH
    +    // when yarn-cluster, all python files can be non-local
    +    if (args.isPython && !isYarnCluster) {
    +      if (Utils.nonLocalPaths(args.primaryResource).nonEmpty) {
    +        SparkSubmit.printErrorAndExit(
    --- End diff --
    
    Also, not a big deal but I actually think this check belongs better in `SparkSubmitArguments`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23856753
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -267,10 +277,22 @@ object SparkSubmit {
         // In yarn-cluster mode, use yarn.Client as a wrapper around the user class
         if (isYarnCluster) {
           childMainClass = "org.apache.spark.deploy.yarn.Client"
    -      if (args.primaryResource != SPARK_INTERNAL) {
    -        childArgs += ("--jar", args.primaryResource)
    +      if (args.isPython) {// yarn-cluster mode for python application
    +      val primaryResourceLocalPath = new Path(args.primaryResource)
    --- End diff --
    
    needs additional indentation


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72377952
  
    Thanks for adding the test and getting it to work @lianhuiwang. Had a few more comments, but this is looking close to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23857948
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -267,10 +277,22 @@ object SparkSubmit {
         // In yarn-cluster mode, use yarn.Client as a wrapper around the user class
         if (isYarnCluster) {
           childMainClass = "org.apache.spark.deploy.yarn.Client"
    -      if (args.primaryResource != SPARK_INTERNAL) {
    -        childArgs += ("--jar", args.primaryResource)
    +      if (args.isPython) {// yarn-cluster mode for python application
    +      val primaryResourceLocalPath = new Path(args.primaryResource)
    +        childArgs += ("--primaryResource", primaryResourceLocalPath.getName)
    +        val pyFilesLocalNames:String = if (args.pyFiles != null) {
    --- End diff --
    
    need a space between the colon and String


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72016569
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26313/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23904474
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -134,12 +136,29 @@ object SparkSubmit {
           }
         }
     
    +    val isYarnCluster = clusterManager == YARN && deployMode == CLUSTER
    +
    +    // Require all python files to be local, so we can add them to the PYTHONPATH
    +    // when yarn-cluster, all python files can be non-local
    +    if (args.isPython && !isYarnCluster) {
    +      if (Utils.nonLocalPaths(args.primaryResource).nonEmpty) {
    +        SparkSubmit.printErrorAndExit(
    +          s"Only local python files are supported: $args.primaryResource")
    +      }
    +      val nonLocalPyFiles = Utils.nonLocalPaths(args.pyFiles).mkString(",")
    +      if (nonLocalPyFiles.nonEmpty) {
    +        SparkSubmit.printErrorAndExit(
    +          s"Only local additional python files are supported: $nonLocalPyFiles")
    +      }
    +    }
    +
         // The following modes are not supported or applicable
         (clusterManager, deployMode) match {
           case (MESOS, CLUSTER) =>
             printErrorAndExit("Cluster deploy mode is currently not supported for Mesos clusters.")
    -      case (_, CLUSTER) if args.isPython =>
    -        printErrorAndExit("Cluster deploy mode is currently not supported for python applications.")
    +      case (STANDALONE, CLUSTER) if args.isPython =>
    +        printErrorAndExit("Cluster deploy mode is currently not supported for python " +
    +          "applications on Standalone clusters")
    --- End diff --
    
    i do not understand what you said. can you tell me the detail? thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-69332863
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25325/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72324906
  
      [Test build #26466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26466/consoleFull) for   PR 3976 at commit [`2adc8f5`](https://github.com/apache/spark/commit/2adc8f591ddd0f253496c18d32b1910d29e04c8d).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-85531984
  
    @tgravescs 
     does this mean that spark has to be installed on all the nodes? That shouldn't be needed on YARN.
    yes, now we need to put spark_home dir to all nodes on YARN. and then set the dir to SPARK_HOME.
    for example you gave, you need to put spark_home dir to /bogus of all nodes on YARN.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #3976: [SPARK-5173]support python application running on yarn cl...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the issue:

    https://github.com/apache/spark/pull/3976
  
    @dileep1236 Can you find pyspark.zip in SPARK_HOME/python/lib/?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72371273
  
      [Test build #26474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26474/consoleFull) for   PR 3976 at commit [`03640ab`](https://github.com/apache/spark/commit/03640ab5c9c9b6890b4b68cf7aaf2183444bd6ed).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-69355776
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25326/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23858127
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -267,10 +277,22 @@ object SparkSubmit {
         // In yarn-cluster mode, use yarn.Client as a wrapper around the user class
         if (isYarnCluster) {
           childMainClass = "org.apache.spark.deploy.yarn.Client"
    -      if (args.primaryResource != SPARK_INTERNAL) {
    -        childArgs += ("--jar", args.primaryResource)
    +      if (args.isPython) {// yarn-cluster mode for python application
    +      val primaryResourceLocalPath = new Path(args.primaryResource)
    +        childArgs += ("--primaryResource", primaryResourceLocalPath.getName)
    +        val pyFilesLocalNames:String = if (args.pyFiles != null) {
    +          args.pyFiles.split(",").map { p => (new Path(p)).getName }.mkString(",")
    +        } else {
    +          "null"
    +        }
    +        childArgs += ("--py-files", pyFilesLocalNames.toString)
    --- End diff --
    
    No need for `toString`, this is already a string.  Also, can we avoid adding this arg at all instead of using "null"?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23861868
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
    @@ -172,7 +172,8 @@ private[spark] class SparkSubmitArguments(args: Seq[String], env: Map[String, St
         }
     
         // Require all python files to be local, so we can add them to the PYTHONPATH
    -    if (isPython) {
    +    // when yarn-cluster, all python files can be non-local
    +    if (isPython && !master.equalsIgnoreCase("yarn-cluster")) {
    --- End diff --
    
    this is not sufficient. Users can manually specify `--master yarn --deploy-mode cluster`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23899881
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -267,10 +292,22 @@ object SparkSubmit {
         // In yarn-cluster mode, use yarn.Client as a wrapper around the user class
         if (isYarnCluster) {
           childMainClass = "org.apache.spark.deploy.yarn.Client"
    -      if (args.primaryResource != SPARK_INTERNAL) {
    -        childArgs += ("--jar", args.primaryResource)
    +      if (args.isPython) {
    +        val mainPyFile = new Path(args.primaryResource).getName
    +        childArgs += ("--primary-py-file", mainPyFile)
    +        val pyFilesNames: String = if (args.pyFiles != null) {
    +          args.pyFiles.split(",").map { p => (new Path(p)).getName }.mkString(",")
    --- End diff --
    
    Use parentheses instead of curly braces here.  Also, probably better to use java.io.File than Path, because Hadoop isn't involved and the file is local.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23857919
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -267,10 +277,22 @@ object SparkSubmit {
         // In yarn-cluster mode, use yarn.Client as a wrapper around the user class
         if (isYarnCluster) {
           childMainClass = "org.apache.spark.deploy.yarn.Client"
    -      if (args.primaryResource != SPARK_INTERNAL) {
    -        childArgs += ("--jar", args.primaryResource)
    +      if (args.isPython) {// yarn-cluster mode for python application
    +      val primaryResourceLocalPath = new Path(args.primaryResource)
    +        childArgs += ("--primaryResource", primaryResourceLocalPath.getName)
    +        val pyFilesLocalNames:String = if (args.pyFiles != null) {
    +          args.pyFiles.split(",").map { p => (new Path(p)).getName }.mkString(",")
    --- End diff --
    
    Nit: map can use parentheses instead of curly braces here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23904882
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -267,10 +292,20 @@ object SparkSubmit {
         // In yarn-cluster mode, use yarn.Client as a wrapper around the user class
         if (isYarnCluster) {
           childMainClass = "org.apache.spark.deploy.yarn.Client"
    -      if (args.primaryResource != SPARK_INTERNAL) {
    -        childArgs += ("--jar", args.primaryResource)
    +      if (args.isPython) {
    +        val mainPyFile = new Path(args.primaryResource).getName
    +        childArgs += ("--primary-py-file", mainPyFile)
    +        if (args.pyFiles != null) {
    +          val pyFilesNames = args.pyFiles.split(",").map(p => (new Path(p)).getName).mkString(",")
    --- End diff --
    
    can you add a comment above this:
    ```
    // These files will be distributed to each machine's working directory, so strip the path prefix
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72174543
  
    @tgravescs  or @andrewor14 Can you help to review this PR? thanks. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-73538842
  
    I thought we had it documented somewhere in what modes different things ran but I'm not seeing it in the documentation.  I think I will file jira to add some documentation so people know what modes they can use to run what things.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23899671
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -134,12 +136,29 @@ object SparkSubmit {
           }
         }
     
    +    val isYarnCluster = clusterManager == YARN && deployMode == CLUSTER
    +
    +    // Require all python files to be local, so we can add them to the PYTHONPATH
    +    // when yarn-cluster, all python files can be non-local
    +    if (args.isPython && !isYarnCluster) {
    +      if (Utils.nonLocalPaths(args.primaryResource).nonEmpty) {
    +        SparkSubmit.printErrorAndExit(
    +          s"Only local python files are supported: $args.primaryResource")
    +      }
    +      val nonLocalPyFiles = Utils.nonLocalPaths(args.pyFiles).mkString(",")
    +      if (nonLocalPyFiles.nonEmpty) {
    +        SparkSubmit.printErrorAndExit(
    +          s"Only local additional python files are supported: $nonLocalPyFiles")
    +      }
    +    }
    +
         // The following modes are not supported or applicable
         (clusterManager, deployMode) match {
           case (MESOS, CLUSTER) =>
             printErrorAndExit("Cluster deploy mode is currently not supported for Mesos clusters.")
    -      case (_, CLUSTER) if args.isPython =>
    -        printErrorAndExit("Cluster deploy mode is currently not supported for python applications.")
    +      case (STANDALONE, CLUSTER) if args.isPython =>
    +        printErrorAndExit("Cluster deploy mode is currently not supported for python " +
    +          "applications on Standalone clusters")
    --- End diff --
    
    Nit: add a period


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72303459
  
      [Test build #26453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26453/consoleFull) for   PR 3976 at commit [`097a5ec`](https://github.com/apache/spark/commit/097a5ec37456bf9d13a952f4108a750b9f9f84d0).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23858269
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -267,10 +277,22 @@ object SparkSubmit {
         // In yarn-cluster mode, use yarn.Client as a wrapper around the user class
         if (isYarnCluster) {
           childMainClass = "org.apache.spark.deploy.yarn.Client"
    -      if (args.primaryResource != SPARK_INTERNAL) {
    -        childArgs += ("--jar", args.primaryResource)
    +      if (args.isPython) {// yarn-cluster mode for python application
    +      val primaryResourceLocalPath = new Path(args.primaryResource)
    +        childArgs += ("--primaryResource", primaryResourceLocalPath.getName)
    +        val pyFilesLocalNames:String = if (args.pyFiles != null) {
    +          args.pyFiles.split(",").map { p => (new Path(p)).getName }.mkString(",")
    --- End diff --
    
    Also, it seems like if the primary resource is a jar, it isn't truncated with getName.  Is there a reason this needs to be different for a python file?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23904612
  
    --- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala ---
    @@ -98,6 +121,9 @@ class YarnClusterSuite extends FunSuite with BeforeAndAfterAll with Matchers wit
         }
     
         fakeSparkJar = File.createTempFile("sparkJar", null, tempDir)
    +    val sparkHome = sys.props.getOrElse("spark.test.home", fail("spark.test.home is not set!"))
    --- End diff --
    
    Because PythonRunner should use sparkPythonPath that contains pyspark library, here i  add SPARK_HOME to env of executor and driver. if we do not add this env, python in executor or driver cannot work. @sryza


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23885383
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -165,6 +168,13 @@ object SparkSubmit {
           }
         }
     
    +    // In yarn-cluster mode for a python app, add primary Resource and pyFiles to files
    +    // that can be distributed with the job
    +    if (args.isPython && deployMode == CLUSTER) {
    --- End diff --
    
    i will add clusterManager == YARN && deployMode == CLUSTER that can says yarn-cluster mode.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-85502310
  
    so it appears setting SPARK_HOME to anything on AM and executors works around this error:
    spark.yarn.appMasterEnv.SPARK_HOME /bogus
    spark.executorEnv.SPARK_HOME /bogus
    
    Is there jira to fix this?  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23861584
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -267,10 +277,22 @@ object SparkSubmit {
         // In yarn-cluster mode, use yarn.Client as a wrapper around the user class
         if (isYarnCluster) {
           childMainClass = "org.apache.spark.deploy.yarn.Client"
    -      if (args.primaryResource != SPARK_INTERNAL) {
    -        childArgs += ("--jar", args.primaryResource)
    +      if (args.isPython) {// yarn-cluster mode for python application
    +      val primaryResourceLocalPath = new Path(args.primaryResource)
    --- End diff --
    
    I would just do
    ```
    val mainPyFile = new Path(args.primaryResource).getName
    ```
    since you don't reference the path itself elsewhere


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23904915
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -134,12 +136,29 @@ object SparkSubmit {
           }
         }
     
    +    val isYarnCluster = clusterManager == YARN && deployMode == CLUSTER
    +
    +    // Require all python files to be local, so we can add them to the PYTHONPATH
    +    // when yarn-cluster, all python files can be non-local
    +    if (args.isPython && !isYarnCluster) {
    +      if (Utils.nonLocalPaths(args.primaryResource).nonEmpty) {
    +        SparkSubmit.printErrorAndExit(
    --- End diff --
    
    If you move this here then you don't need the `SparkSubmit.printErrorAndExit`. You can just do `printErrorAndExit`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72403015
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26482/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #3976: [SPARK-5173]support python application running on yarn cl...

Posted by dileep1236 <gi...@git.apache.org>.
Github user dileep1236 commented on the issue:

    https://github.com/apache/spark/pull/3976
  
    When we run , we are getting the error like 
    _java.io.FileNotFoundException: File does not exist: hdfs.../pyspark.zip_. What would be the reason. or 
    I have tried the below command
     spark-submit --master yarn-cluster --num-executors 4 --driver-memory 1g --executor-memory 1g main.py



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/3976


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72321234
  
      [Test build #26464 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26464/consoleFull) for   PR 3976 at commit [`d60bc60`](https://github.com/apache/spark/commit/d60bc6069cf65637622472ef1cd27153333df53c).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-71034833
  
    yes, this PR is for batch mode by py file. so i think yarn-client mode is enough for  interactive /bin/pyspark. but sometime batch python application need to run on yarn-cluster mode, just like scala or java.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72302619
  
      [Test build #26452 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26452/consoleFull) for   PR 3976 at commit [`905a106`](https://github.com/apache/spark/commit/905a10610532578c774e58d12b927597330fb9ff).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72403008
  
      [Test build #26482 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26482/consoleFull) for   PR 3976 at commit [`0319ae3`](https://github.com/apache/spark/commit/0319ae328b2db694684ea586cbb7d49fb2b487c7).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72410777
  
      [Test build #26490 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26490/consoleFull) for   PR 3976 at commit [`67f8cee`](https://github.com/apache/spark/commit/67f8cee9e25b5bd05c0252705b1f67cb63b0fa01).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23885537
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
    @@ -172,7 +172,8 @@ private[spark] class SparkSubmitArguments(args: Seq[String], env: Map[String, St
         }
     
         // Require all python files to be local, so we can add them to the PYTHONPATH
    -    if (isPython) {
    +    // when yarn-cluster, all python files can be non-local
    +    if (isPython && !master.equalsIgnoreCase("yarn-cluster")) {
    --- End diff --
    
    how about i move checkRequired here to SparkSubmit, because SparkSubmit can get isYarnClusterMode.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72348373
  
      [Test build #26470 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26470/consoleFull) for   PR 3976 at commit [`47d2fc3`](https://github.com/apache/spark/commit/47d2fc35e53a8851790607085bc67e94736358d6).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72323941
  
    @JoshRosen can you help me? now i add a unit test for python applicaiton on yarn cluster mode. but now there is a failed. i think its reason is environment  of pyspark are not set. can you tell me how to set  pyspark's environment in my java unit test.thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3976#discussion_r23904808
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -134,12 +136,29 @@ object SparkSubmit {
           }
         }
     
    +    val isYarnCluster = clusterManager == YARN && deployMode == CLUSTER
    +
    +    // Require all python files to be local, so we can add them to the PYTHONPATH
    +    // when yarn-cluster, all python files can be non-local
    +    if (args.isPython && !isYarnCluster) {
    +      if (Utils.nonLocalPaths(args.primaryResource).nonEmpty) {
    +        SparkSubmit.printErrorAndExit(
    +          s"Only local python files are supported: $args.primaryResource")
    +      }
    +      val nonLocalPyFiles = Utils.nonLocalPaths(args.pyFiles).mkString(",")
    +      if (nonLocalPyFiles.nonEmpty) {
    +        SparkSubmit.printErrorAndExit(
    +          s"Only local additional python files are supported: $nonLocalPyFiles")
    +      }
    +    }
    +
         // The following modes are not supported or applicable
         (clusterManager, deployMode) match {
           case (MESOS, CLUSTER) =>
             printErrorAndExit("Cluster deploy mode is currently not supported for Mesos clusters.")
    -      case (_, CLUSTER) if args.isPython =>
    -        printErrorAndExit("Cluster deploy mode is currently not supported for python applications.")
    +      case (STANDALONE, CLUSTER) if args.isPython =>
    +        printErrorAndExit("Cluster deploy mode is currently not supported for python " +
    +          "applications on Standalone clusters")
    --- End diff --
    
    he means add a `.` character at the end of the string, like
    `"applications on Standalone clusters."`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72317446
  
      [Test build #26462 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26462/consoleFull) for   PR 3976 at commit [`5b30064`](https://github.com/apache/spark/commit/5b300648fe53d9de604e8afce7580fddfe6bbaef).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72399690
  
    @lianhuiwang what happens now if the submission node uses a different SPARK_HOME from the machines? Does it still work?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72319231
  
      [Test build #26464 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26464/consoleFull) for   PR 3976 at commit [`d60bc60`](https://github.com/apache/spark/commit/d60bc6069cf65637622472ef1cd27153333df53c).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72317452
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26462/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5173]support python application running...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3976#issuecomment-72327239
  
      [Test build #26466 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26466/consoleFull) for   PR 3976 at commit [`2adc8f5`](https://github.com/apache/spark/commit/2adc8f591ddd0f253496c18d32b1910d29e04c8d).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org