[jira] [Commented] (SPARK-1455) Determine which test suites to run based on code changes

["Nicholas Chammas (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14134856#comment-14134856 ] 

Nicholas Chammas commented on SPARK-1455:
-----------------------------------------

I think we could reuse some of [the logic here|https://github.com/databricks/spark-pr-dashboard/blob/bf289c15ffd959eed0911f12591bb1b08672b2dd/sparkprs/models.py#L71] to determine what tests to run from the files that were changed.

> Determine which test suites to run based on code changes
> --------------------------------------------------------
>
>                 Key: SPARK-1455
>                 URL: https://issues.apache.org/jira/browse/SPARK-1455
>             Project: Spark
>          Issue Type: Improvement
>          Components: Project Infra
>            Reporter: Patrick Wendell
>             Fix For: 1.2.0
>
>
> Right now we run the entire set of tests for every change. This means the tests take a long time. Our pull request builder checks out the merge branch from git, so we could do a diff and figure out what source files were changed, and run a more isolated set of tests. We should just run tests in a way that reflects the inter-dependencies of the project. E.g:
> - If Spark core is modified, we should run all tests
> - If just SQL is modified, we should run only the SQL tests
> - If just Streaming is modified, we should run only the streaming tests
> - If just Pyspark is modified, we only run the PySpark tests.
> And so on. I think this would reduce the RTT of the tests a lot and it should be pretty easy to accomplish with some scripting foo.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org