You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Saikat Kanjilal (JIRA)" <ji...@apache.org> on 2016/11/09 19:22:59 UTC
[jira] [Comment Edited] (SPARK-9487) Use the same num. worker
threads in Scala/Python unit tests
[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15651806#comment-15651806 ]
Saikat Kanjilal edited comment on SPARK-9487 at 11/9/16 7:22 PM:
-----------------------------------------------------------------
Ok , for some odd reason my local branch had the changes but weren't committed, PR is here: https://github.com/skanjila/spark/commit/ec0b2a81dc8362e84e70457873560d997a7cb244
I added the change to local[4] to both streaming as well as repl, based on what I'm seeing locally all Java/Scala changes should be accounted for and unit tests pass, the only exception is the code inside spark examples in PageViewStream.scala, should I change this, seems like it doesn't belong as part of the unit tests.
My next TODOs:
1) Change the example code if it makes sense in PageViewStream
2) Start the code changes to fix the python unit tests
Let me know thoughts or concerns.
was (Author: kanjilal):
Ok , for some odd reason my local branch had the changes but weren't committed, PR is here: https://github.com/skanjila/spark/commit/ec0b2a81dc8362e84e70457873560d997a7cb244
I added the change to local[4] to both streaming as well as repl, based on what I'm seeing locally all Java/Scala changes should be accounted for except for spark examples with the code inside PageViewStream.scala, should I change this, seems like it doesn't belong as part of the unit tests.
My next TODOs:
1) Change the example code if it makes sense in PageViewStream
2) Start the code changes to fix the python unit tests
Let me know thoughts or concerns.
> Use the same num. worker threads in Scala/Python unit tests
> -----------------------------------------------------------
>
> Key: SPARK-9487
> URL: https://issues.apache.org/jira/browse/SPARK-9487
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, Spark Core, SQL, Tests
> Affects Versions: 1.5.0
> Reporter: Xiangrui Meng
> Labels: starter
> Attachments: ContextCleanerSuiteResults, HeartbeatReceiverSuiteResults
>
>
> In Python we use `local[4]` for unit tests, while in Scala/Java we use `local[2]` and `local` for some unit tests in SQL, MLLib, and other components. If the operation depends on partition IDs, e.g., random number generator, this will lead to different result in Python and Scala/Java. It would be nice to use the same number in all unit tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org