[jira] [Created] (SPARK-3534) Avoid running MLlib and Streaming tests when testing SQL PRs

["Michael Armbrust (JIRA)" <ji...@apache.org>]

Michael Armbrust created SPARK-3534:
---------------------------------------

             Summary: Avoid running MLlib and Streaming tests when testing SQL PRs
                 Key: SPARK-3534
                 URL: https://issues.apache.org/jira/browse/SPARK-3534
             Project: Spark
          Issue Type: Bug
          Components: Project Infra, SQL
            Reporter: Michael Armbrust
            Priority: Blocker


We are bumping up against the 120 minute time limit for tests pretty regularly now.  Since we have decreased the number of shuffle partitions and up-ed the parallelism I don't think there is much low hanging fruit to speed up the SQL tests. (The tests that are listed as taking 2-3 minutes are actually 100s of tests that I think are valuable).  Instead I propose we avoid running tests that we don't need to.

This will have the added benefit of eliminating failures in SQL due to flaky streaming tests.

Note that this won't fix the full builds that are run for every commit.  There I think we just just up the test timeout.

cc: [~joshrosen] [~pwendell]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org