[jira] [Comment Edited] (SPARK-12243) PySpark tests are slow in Jenkins

["Dongjoon Hyun (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-12243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182460#comment-15182460 ] 

Dongjoon Hyun edited comment on SPARK-12243 at 3/7/16 1:49 AM:
---------------------------------------------------------------

According to the log, the total time of all tests are *3077s*. 
So, the minimum required time for 4 processes is *769s*.



was (Author: dongjoon):
According to the log, the total time of all tests are **3077s**. 
So, the minimum required time for 4 processes is **769s**.


> PySpark tests are slow in Jenkins
> ---------------------------------
>
>                 Key: SPARK-12243
>                 URL: https://issues.apache.org/jira/browse/SPARK-12243
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Project Infra, PySpark, Tests
>            Reporter: Josh Rosen
>
> In the Jenkins pull request builder, it looks like PySpark tests take around 992 seconds (~16.5 minutes) of end-to-end time to run, despite the fact that we run four Python test suites in parallel. We should try to figure out why this is slow and see if there's any easy way to speed things up.
> Note that the PySpark streaming tests take about 5 minutes to run, so best-case we're looking at a 10 minute speedup via further parallelization. We should also try to see whether there are individual slow tests in those Python suites which can be sped up or skipped.
> We could also consider running only the Python 2.6 tests in non-Pyspark pull request builds and reserve testing of all Python versions for builds which touch PySpark-related code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org