You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Marcelo Vanzin (JIRA)" <ji...@apache.org> on 2019/02/19 21:18:00 UTC

[jira] [Created] (SPARK-26934) python dependencies with "local:" URIs are not visible to executors

Marcelo Vanzin created SPARK-26934:
--------------------------------------

             Summary: python dependencies with "local:" URIs are not visible to executors
                 Key: SPARK-26934
                 URL: https://issues.apache.org/jira/browse/SPARK-26934
             Project: Spark
          Issue Type: Bug
          Components: PySpark, Spark Core
    Affects Versions: 2.4.0
            Reporter: Marcelo Vanzin


Noticed this when running tests for SPARK-24736. If you provide a file with a "local:" URI to {{--py-files}}, spark-submit makes it visible to the driver, but not executors. That makes code that runs in executors and references those files fail.

The underlying issue is that when providing dependency information to executors, when running tasks, {{SparkContext}} erases any references to files with a "local:" scheme, so executors don't know they exist.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org