[jira] [Resolved] (SPARK-1917) PySpark fails to import functions from {{scipy.special}}

["Matei Zaharia (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matei Zaharia resolved SPARK-1917.
----------------------------------

    Resolution: Fixed

> PySpark fails to import functions from {{scipy.special}}
> --------------------------------------------------------
>
>                 Key: SPARK-1917
>                 URL: https://issues.apache.org/jira/browse/SPARK-1917
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 0.9.0, 1.0.0
>            Reporter: Uri Laserson
>            Assignee: Uri Laserson
>             Fix For: 0.9.2, 1.0.1
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> PySpark is able to load {{numpy}} functions, but not {{scipy.special}} functions.  For example  take this snippet:
> {code}
> from numpy import exp
> from scipy.special import gammaln
> a = range(1, 11)
> b = sc.parallelize(a)
> c = b.map(exp)
> d = b.map(special.gammaln)
> {code}
> Calling {{c.collect()}} will return the expected result.  However, calling {{d.collect()}} will fail with
> {code}
> KeyError: (('gammaln',), <function _getobject at 0x10c0879b0>, ('scipy.special', 'gammaln'))
> {code}
> in {{cloudpickle.py}} module in {{_getobject}}.
> The reason is that {{_getobject}} executes {{__import__(modname)}}, which only loads the top-level package {{X}} in case {{modname}} is like {{X.Y}}.  It is failing because {{gammaln}} is not a member of {{scipy}}.  The fix (for which I will shortly submit a PR) is to add {{fromlist=[attribute]}} to the {{__import__}} call, which will load the innermost module.
> See 
> [https://docs.python.org/2/library/functions.html#__import__]
> and
> [http://stackoverflow.com/questions/9544331/from-a-b-import-x-using-import]



--
This message was sent by Atlassian JIRA
(v6.2#6252)