[jira] [Updated] (SPARK-4477) remove numpy from RDDSampler of PySpark

["Davies Liu (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Davies Liu updated SPARK-4477:
------------------------------
    Affects Version/s: 1.2.0
                       0.8.1
                       0.9.1
                       1.0.2
                       1.1.0

> remove numpy from RDDSampler of PySpark
> ---------------------------------------
>
>                 Key: SPARK-4477
>                 URL: https://issues.apache.org/jira/browse/SPARK-4477
>             Project: Spark
>          Issue Type: New Feature
>          Components: PySpark
>    Affects Versions: 0.8.1, 0.9.1, 1.0.2, 1.1.0, 1.2.0
>            Reporter: Davies Liu
>
> In RDDSampler, it try use numpy to gain better performance for possion(), but the number of call of random() is only (1+faction) * N in the pure python implementation of possion(), so there is no much performance gain from numpy. 
> numpy is not a dependent of pyspark, so it maybe introduce some problem, such as there is no numpy installed in slaves, but only installed master, as reported in xxxx. 
> It also complicate the code a lot, so we may should remove numpy from RDDSampler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org