[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

["Evan Chen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941961#comment-14941961 ] 

Evan Chen commented on SPARK-9487:
----------------------------------

Hey Xiangrui,

What would be the preferred num. worker threads? Should we set all of them to local[2] to stay consistent with the Scala/Java side?

Thanks

> Use the same num. worker threads in Scala/Python unit tests
> -----------------------------------------------------------
>
>                 Key: SPARK-9487
>                 URL: https://issues.apache.org/jira/browse/SPARK-9487
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark, Spark Core, SQL, Tests
>    Affects Versions: 1.5.0
>            Reporter: Xiangrui Meng
>
> In Python we use `local[4]` for unit tests, while in Scala/Java we use `local[2]` and `local` for some unit tests in SQL, MLLib, and other components. If the operation depends on partition IDs, e.g., random number generator, this will lead to different result in Python and Scala/Java. It would be nice to use the same number in all unit tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org