You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2022/06/20 00:54:00 UTC

[jira] [Updated] (SPARK-39515) Improve scheduled jobs in GitHub Actions

     [ https://issues.apache.org/jira/browse/SPARK-39515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon updated SPARK-39515:
---------------------------------
    Description: 
There are four problems to address.

*First*, the scheduled jobs are broken as below:

https://github.com/apache/spark/actions/runs/2513261706
https://github.com/apache/spark/actions/runs/2512750310
https://github.com/apache/spark/actions/runs/2509238648
https://github.com/apache/spark/actions/runs/2508246903
https://github.com/apache/spark/actions/runs/2507327914
https://github.com/apache/spark/actions/runs/2506654808
https://github.com/apache/spark/actions/runs/2506143939
https://github.com/apache/spark/actions/runs/2502449498
https://github.com/apache/spark/actions/runs/2501400490
https://github.com/apache/spark/actions/runs/2500407628
https://github.com/apache/spark/actions/runs/2499722093
https://github.com/apache/spark/actions/runs/2499196539
https://github.com/apache/spark/actions/runs/2496544415
https://github.com/apache/spark/actions/runs/2495444227
https://github.com/apache/spark/actions/runs/2493402272
https://github.com/apache/spark/actions/runs/2492759618
https://github.com/apache/spark/actions/runs/2492227816

See also https://github.com/apache/spark/pull/36899 or https://github.com/apache/spark/pull/36890

*Second*, it is very difficult to navigate scheduled jobs now. We should use https://github.com/apache/spark/actions/workflows/build_and_test.yml?query=event%3Aschedule link and manually search one by one.

Since GitHub added the feature to import other workflow, we should leverage this feature, see also https://github.com/apache/spark/blob/master/.github/workflows/build_and_test_ansi.yml. Once we can separate them, it will be defined as a separate workflows. We might have to clean up existing build history to make it easier to read (it requires committer-only permission to clean them up).

*Third*, we should set the scheduled jobs for branch-3.3, see also https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L78-L83 for branch-3.2 job.

*Forth*, we should improve duplicated test skipping logic. See also https://github.com/apache/spark/pull/36413#issuecomment-1157205469 and https://github.com/apache/spark/pull/36888

  was:
There are four problems to address.

First, the scheduled jobs are broken as below:

https://github.com/apache/spark/actions/runs/2513261706
https://github.com/apache/spark/actions/runs/2512750310
https://github.com/apache/spark/actions/runs/2509238648
https://github.com/apache/spark/actions/runs/2508246903
https://github.com/apache/spark/actions/runs/2507327914
https://github.com/apache/spark/actions/runs/2506654808
https://github.com/apache/spark/actions/runs/2506143939
https://github.com/apache/spark/actions/runs/2502449498
https://github.com/apache/spark/actions/runs/2501400490
https://github.com/apache/spark/actions/runs/2500407628
https://github.com/apache/spark/actions/runs/2499722093
https://github.com/apache/spark/actions/runs/2499196539
https://github.com/apache/spark/actions/runs/2496544415
https://github.com/apache/spark/actions/runs/2495444227
https://github.com/apache/spark/actions/runs/2493402272
https://github.com/apache/spark/actions/runs/2492759618
https://github.com/apache/spark/actions/runs/2492227816

See also https://github.com/apache/spark/pull/36899 or https://github.com/apache/spark/pull/36890

Second, it is very difficult to navigate scheduled jobs now. We should use https://github.com/apache/spark/actions/workflows/build_and_test.yml?query=event%3Aschedule link and manually search one by one.

Since GitHub added the feature to import other workflow, we should leverage this feature, see also https://github.com/apache/spark/blob/master/.github/workflows/build_and_test_ansi.yml. Once we can separate them, it will be defined as a separate workflows. We might have to clean up existing build history to make it easier to read (it requires committer-only permission to clean them up).

Third, we should set the scheduled jobs for branch-3.3, see also https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L78-L83 for branch-3.2 job.

Forth, we should improve duplicated test skipping logic. See also https://github.com/apache/spark/pull/36413#issuecomment-1157205469 and https://github.com/apache/spark/pull/36888


> Improve scheduled jobs in GitHub Actions
> ----------------------------------------
>
>                 Key: SPARK-39515
>                 URL: https://issues.apache.org/jira/browse/SPARK-39515
>             Project: Spark
>          Issue Type: Umbrella
>          Components: Build, Project Infra
>    Affects Versions: 3.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Critical
>
> There are four problems to address.
> *First*, the scheduled jobs are broken as below:
> https://github.com/apache/spark/actions/runs/2513261706
> https://github.com/apache/spark/actions/runs/2512750310
> https://github.com/apache/spark/actions/runs/2509238648
> https://github.com/apache/spark/actions/runs/2508246903
> https://github.com/apache/spark/actions/runs/2507327914
> https://github.com/apache/spark/actions/runs/2506654808
> https://github.com/apache/spark/actions/runs/2506143939
> https://github.com/apache/spark/actions/runs/2502449498
> https://github.com/apache/spark/actions/runs/2501400490
> https://github.com/apache/spark/actions/runs/2500407628
> https://github.com/apache/spark/actions/runs/2499722093
> https://github.com/apache/spark/actions/runs/2499196539
> https://github.com/apache/spark/actions/runs/2496544415
> https://github.com/apache/spark/actions/runs/2495444227
> https://github.com/apache/spark/actions/runs/2493402272
> https://github.com/apache/spark/actions/runs/2492759618
> https://github.com/apache/spark/actions/runs/2492227816
> See also https://github.com/apache/spark/pull/36899 or https://github.com/apache/spark/pull/36890
> *Second*, it is very difficult to navigate scheduled jobs now. We should use https://github.com/apache/spark/actions/workflows/build_and_test.yml?query=event%3Aschedule link and manually search one by one.
> Since GitHub added the feature to import other workflow, we should leverage this feature, see also https://github.com/apache/spark/blob/master/.github/workflows/build_and_test_ansi.yml. Once we can separate them, it will be defined as a separate workflows. We might have to clean up existing build history to make it easier to read (it requires committer-only permission to clean them up).
> *Third*, we should set the scheduled jobs for branch-3.3, see also https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L78-L83 for branch-3.2 job.
> *Forth*, we should improve duplicated test skipping logic. See also https://github.com/apache/spark/pull/36413#issuecomment-1157205469 and https://github.com/apache/spark/pull/36888



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org