[jira] [Created] (SPARK-3634) Python modules added through addPyFile should take precedence over system modules

["Josh Rosen (JIRA)" <ji...@apache.org>]

Josh Rosen created SPARK-3634:
---------------------------------

             Summary: Python modules added through addPyFile should take precedence over system modules
                 Key: SPARK-3634
                 URL: https://issues.apache.org/jira/browse/SPARK-3634
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 1.1.0, 1.0.2
            Reporter: Josh Rosen


Python modules added through {{SparkContext.addPyFile()}} are currently _appended_ to {{sys.path}}; this is probably the opposite of the behavior that we want, since it causes system versions of modules to take precedence over versions explicitly added by users.

To fix this, we should change the {{sys.path}} manipulation code in {{context.py}} and {{worker.py}} to prepend files to {{sys.path}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org