You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dhruve Ashar (JIRA)" <ji...@apache.org> on 2019/02/05 18:09:00 UTC
[jira] [Comment Edited] (SPARK-26827) Support importing python
modules having shared objects(.so)
[ https://issues.apache.org/jira/browse/SPARK-26827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761072#comment-16761072 ]
Dhruve Ashar edited comment on SPARK-26827 at 2/5/19 6:08 PM:
--------------------------------------------------------------
Resolution : Pass the same archive with py-files and archives option.
was (Author: dhruve ashar):
Pass the same archive with py-files and archives option.
> Support importing python modules having shared objects(.so)
> -----------------------------------------------------------
>
> Key: SPARK-26827
> URL: https://issues.apache.org/jira/browse/SPARK-26827
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
> Affects Versions: 2.3.2, 2.4.0
> Reporter: Dhruve Ashar
> Priority: Major
>
> If a user wants to import dynamic modules, specifically having .so files, this is currently disallowed by python from a zip file. ([https://docs.python.org/3/library/zipimport.html)] and currently spark doesn't support this either.
> Files which are passed using py-files options are placed on the PYTHONPATH, but are not extracted. While files which are passed as archives, are extracted but not placed on the PYTHONPATH. The dynamic modules can be loaded if they are extracted and added to the PYTHONPATH.
>
> Has anyone encountered this issue before and what is the best way to go about it?
>
> Some possible solutions:
> 1 - Get around this issue, by passing the archive with py-files and archives option, this extracts the archive as well as adds it to the path. Gotcha - both have to be named the same. I have tested this and it works, but its just a workaround.
> 2 - We add a new config like py-archives which takes all the files and extracts them and also adds them to the PYTHONPATH. Or just examine the contents of the zip file and if it has dynamic modules then do the same. I am happy to work on the fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org