[jira] [Commented] (SPARK-54364) Separate test_install_spark out from our CI

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-54364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18038566#comment-18038566 ] 

Dongjoon Hyun commented on SPARK-54364:
---------------------------------------

Sounds good to me.

> Separate test_install_spark out from our CI
> -------------------------------------------
>
>                 Key: SPARK-54364
>                 URL: https://issues.apache.org/jira/browse/SPARK-54364
>             Project: Spark
>          Issue Type: Test
>          Components: PySpark
>    Affects Versions: 4.1.0
>            Reporter: Tian Gao
>            Priority: Minor
>
> test_install_spark can take 2000~3000s based on the network, but it's not testing any local code. It basically download the spark from remote and try to unpack it. Now it's part of pyspark-core which means any commit that changes pyspark core will trigger this test in our CI. e.g.
> [https://github.com/apache/spark/actions/runs/19347901449/job/55352935128]
> This wastes a lot of our limited CI resources.
> I think we should at least disable this on main commit triggered CI - so leave it only for scheduled CIs.
> If we are confident about how this works, we can have a single CI that run this test once per day.
> I think we are probably spending more than 10 hours every day to run this test which does not give us any meaningful coverage.
> [~dongjoon] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org