You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Xiangrui Meng (JIRA)" <ji...@apache.org> on 2015/07/30 23:25:05 UTC

[jira] [Created] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

Xiangrui Meng created SPARK-9487:
------------------------------------

             Summary: Use the same num. worker threads in Scala/Python unit tests
                 Key: SPARK-9487
                 URL: https://issues.apache.org/jira/browse/SPARK-9487
             Project: Spark
          Issue Type: Improvement
          Components: PySpark, Spark Core, SQL, Tests
    Affects Versions: 1.5.0
            Reporter: Xiangrui Meng


In Python we use `local[4]` for unit tests, while in Scala/Java we use `local[2]` and `local` for some unit tests in SQL, MLLib, and other components. If the operation depends on partition IDs, e.g., random number generator, this will lead to different result in Python and Scala/Java. It would be nice to use the same number in all unit tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org