You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by skonto <gi...@git.apache.org> on 2017/07/10 10:11:43 UTC

[GitHub] spark pull request #18587: [SPARK-12559][mesos] fix --packages for mesos

GitHub user skonto opened a pull request:

    https://github.com/apache/spark/pull/18587

    [SPARK-12559][mesos] fix --packages for mesos

    ## What changes were proposed in this pull request?
    Fixes --packages flag for mesos in cluster mode. Probably I will handle standalone and Yarn in another commit, I need to investigate those cases as they are different.
    
    ## How was this patch tested?
    Tested with a community 1.9 dc/os cluster. packages were successfully resolved in cluster mode within a container.
    
    @andrewor14  @susanxhuynh @ArtRand  pls review.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/skonto/spark fix_packages_mesos_cluster

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/18587.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #18587
    
----
commit 6139631bc6b0a2bf773ae2a3e90899dfa45fd6e2
Author: Stavros Kontopoulos <st...@gmail.com>
Date:   2017-07-10T10:03:20Z

    fix --packages for mesos

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18587: [SPARK-12559][Mesos] fix --packages for mesos

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18587
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79462/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18587: [SPARK-21403][Mesos] fix --packages for mesos

Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on the issue:

    https://github.com/apache/spark/pull/18587
  
    thnx @vanzin I am working on some code for the standalone case. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18587: [SPARK-12559][Mesos] fix --packages for mesos

Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on the issue:

    https://github.com/apache/spark/pull/18587
  
    @vanzin 
    > args.jars = mergeFileLists(args.jars, resolvedMavenCoordinates)
    
    My understanding:
    Files intended for the appmaster are handled by copying to hdfs:
    https://github.com/apache/spark/blob/1cad31f00644d899d8e74d58c6eb4e9f72065473/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L365
    The distributed cache manager does the work by utilizing the distributed Hadoop cache for jars at the executor side. 
    https://github.com/apache/spark/blob/ab9872db1f9c0f289541ec5756d1a142d85545ce/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientDistributedCacheManager.scala
    
    > As for the change, it seems to work because the Mesos backend starts the driver using spark-submit, right?
    That is the idea. The second time the submit is called in order to launch the driver, the submit is done using client mode which is the default. The call for launching the driver in client mode is here:
    https://github.com/apache/spark/blob/8da3f7041aafa71d7596b531625edb899970fec2/resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L411-L442
    
    > If that's the case it seems fine, although it kinda loses the ability to use the ivy cache on the machine launching the job...
    
    That's is correct. If you launch jobs from a machine within the same network cache makes sense. Otherwise it is just copying over the internet. Now in restarts it is also important when you re-launch something to access it from the cache. In spark there is no functionality for the mesos code to exploit any type of cache from what I see. I would prefer a unified cluster layer for certain things like in the case of other frameworks: https://issues.apache.org/jira/browse/FLINK-6177
    I guess some refactoring would make yarn stuff accessible from the mesos part in Spark. Some other options are the fetcher cache mechanism but not so sure if it is distributed, it is only local per agent, need to discuss it further with @susanxhuynh @ArtRand
    
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18587: [SPARK-12559][Mesos] fix --packages for mesos

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/18587
  
    I'm 99% sure there's nothing to do for YARN. This line takes care of it:
    
        args.jars = mergeFileLists(args.jars, resolvedMavenCoordinates)
    
    YARN cluster mode will distribute all jars in `args.jars` to the app.
    
    As for the change, it seems to work because the Mesos backend starts the driver using `spark-submit`, right? (It would probably have to change the deploy mode from "cluster" to "client" when doing that but I didn't dig that much into the code...)
    
    If that's the case it seems fine, although it kinda loses the ability to use the ivy cache on the machine launching the job...
    
    Also, I'd be more comfortable if someone more familiar with the Mesos backend could take a look. Not sure who that person is.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18587: [SPARK-12559][Mesos] fix --packages for mesos

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18587
  
    **[Test build #79462 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79462/testReport)** for PR 18587 at commit [`6139631`](https://github.com/apache/spark/commit/6139631bc6b0a2bf773ae2a3e90899dfa45fd6e2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18587: [SPARK-12559][Mesos] fix --packages for mesos

Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18587#discussion_r126382374
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
    @@ -269,6 +269,25 @@ object SparkSubmit extends CommandLineUtils {
           }
         }
     
    +    // Fail fast, the following modes are not supported or applicable
    +    (clusterManager, deployMode) match {
    +      case (STANDALONE, CLUSTER) if args.isPython =>
    +        printErrorAndExit("Cluster deploy mode is currently not supported for python " +
    +          "applications on standalone clusters.")
    +      case (STANDALONE, CLUSTER) if args.isR =>
    +        printErrorAndExit("Cluster deploy mode is currently not supported for R " +
    +          "applications on standalone clusters.")
    +      case (LOCAL, CLUSTER) =>
    +        printErrorAndExit("Cluster deploy mode is not compatible with master \"local\"")
    +      case (_, CLUSTER) if isShell(args.primaryResource) =>
    +        printErrorAndExit("Cluster deploy mode is not applicable to Spark shells.")
    +      case (_, CLUSTER) if isSqlShell(args.mainClass) =>
    +        printErrorAndExit("Cluster deploy mode is not applicable to Spark SQL shell.")
    +      case (_, CLUSTER) if isThriftServer(args.mainClass) =>
    +        printErrorAndExit("Cluster deploy mode is not applicable to Spark Thrift server.")
    +      case _ =>
    +    }
    +
    --- End diff --
    
    This is for failing faster no need to resolve dependencies if we are going to fail.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18587: [SPARK-21403][Mesos] fix --packages for mesos

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/18587
  
    Sounds good. Merging to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18587: [SPARK-12559][Mesos] fix --packages for mesos

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18587
  
    **[Test build #79462 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79462/testReport)** for PR 18587 at commit [`6139631`](https://github.com/apache/spark/commit/6139631bc6b0a2bf773ae2a3e90899dfa45fd6e2).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #18587: [SPARK-12559][Mesos] fix --packages for mesos

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18587
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #18587: [SPARK-21403][Mesos] fix --packages for mesos

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/18587


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org