[jira] [Resolved] (SPARK-12243) PySpark tests are slow in Jenkins

["Josh Rosen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-12243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Josh Rosen resolved SPARK-12243.
--------------------------------
       Resolution: Fixed
    Fix Version/s: 2.0.0

Issue resolved by pull request 11551
[https://github.com/apache/spark/pull/11551]

> PySpark tests are slow in Jenkins
> ---------------------------------
>
>                 Key: SPARK-12243
>                 URL: https://issues.apache.org/jira/browse/SPARK-12243
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Project Infra, PySpark, Tests
>            Reporter: Josh Rosen
>            Assignee: Dongjoon Hyun
>             Fix For: 2.0.0
>
>
> In the Jenkins pull request builder, it looks like PySpark tests take around 992 seconds (~16.5 minutes) of end-to-end time to run, despite the fact that we run four Python test suites in parallel. We should try to figure out why this is slow and see if there's any easy way to speed things up.
> Note that the PySpark streaming tests take about 5 minutes to run, so best-case we're looking at a 10 minute speedup via further parallelization. We should also try to see whether there are individual slow tests in those Python suites which can be sped up or skipped.
> We could also consider running only the Python 2.6 tests in non-Pyspark pull request builds and reserve testing of all Python versions for builds which touch PySpark-related code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org