You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Xiangrui Meng (JIRA)" <ji...@apache.org> on 2014/05/25 23:06:01 UTC

[jira] [Created] (SPARK-1924) Make local:/ scheme work in more deploy modes

Xiangrui Meng created SPARK-1924:
------------------------------------

             Summary: Make local:/ scheme work in more deploy modes
                 Key: SPARK-1924
                 URL: https://issues.apache.org/jira/browse/SPARK-1924
             Project: Spark
          Issue Type: Sub-task
          Components: Deploy
    Affects Versions: 1.0.0
            Reporter: Xiangrui Meng
            Priority: Minor


A resource marked "local:/" is assumed to be available on the local file system of every node. In such case, no data copying over network should happen. I tested different deploy modes in v1.0, right now we only support local:/ scheme for the app jar and secondary jars in the following modes:

1) local (jars are copied to the working directory)
2) standalone client
3) yarn client

It doesn’t work for --files and python apps (--py-files and app.py). For the next release, we could support more deploy modes.



--
This message was sent by Atlassian JIRA
(v6.2#6252)