[jira] [Updated] (SPARK-58538) Add branch-4.3 CI scheduler and release integration

["Jungtaek Lim (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim updated SPARK-58538:
---------------------------------
    Affects Version/s: 4.3.0
                           (was: 5.0.0)

> Add branch-4.3 CI scheduler and release integration
> ---------------------------------------------------
>
>                 Key: SPARK-58538
>                 URL: https://issues.apache.org/jira/browse/SPARK-58538
>             Project: Spark
>          Issue Type: Improvement
>          Components: Project Infra
>    Affects Versions: 4.3.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> {{branch-4.3}} was cut from {{branch-4.x}} for the 4.3.0 release, but the CI and release plumbing that {{master}} maintains for each active release branch has not been added for it yet. This mirrors SPARK-56710 (which set up {{branch-4.2}} and {{branch-4.x}}) and SPARK-54143 (which set up {{branch-4.1}}).
> h3. Background
> The per-branch build workflows live on the branch itself, and {{branch-4.3}} already inherited a full set of them from {{branch-4.x}} when it was cut ({{build_maven.yml}}, {{build_java17/21/25.yml}}, {{build_non_ansi.yml}}, {{build_python_*.yml}}, etc.). What is missing is on {{master}}: the scheduler that dispatches those workflows against {{--ref branch-4.3}}, plus the places that enumerate active branches.
> h3. Work items
> On {{master}}:
> * Add {{.github/workflows/branch43_scheduler.yml}}, modelled on {{branch42_scheduler.yml}}: cron entries plus a {{workflow_dispatch}} {{target}} choice, dispatching each build workflow with {{gh workflow run <workflow>.yml --repo ${{ github.repository }} --ref branch-4.3}}. Cron times should be staggered against the existing schedulers to spread runner load.
> * Add {{branch-4.3}} to the branch list in {{.github/workflows/publish_snapshot.yml}} (both the {{workflow_dispatch}} default and the matrix default), which is currently {{'["master", "branch-4.x", "branch-4.2", "branch-4.1", "branch-4.0", "branch-3.5"]'}}.
> * Add a {{branch-4.3}} badge row to {{README.md}}, alongside the existing {{branch-4.x}} / {{branch-4.2}} / {{branch-4.1}} rows.
> * Check whether {{dev/create-release/release-util.sh}} needs any update; it currently excludes {{master}} and {{branch-4.x}} when enumerating release branches, and {{branch-4.3}} should be picked up automatically, but this is worth confirming.
> On {{branch-4.3}}:
> * The inherited workflows are still titled for {{branch-4.x}} — e.g. {{build_maven.yml}} reads {{name: "Build / Maven (branch-4.x, Scala 2.13, Hadoop 3, JDK 17)"}} and {{build_java17.yml}} reads {{name: "Build / Java17 (branch-4.x, ...)"}}. These names appear in the Actions UI and check-run names, so they should be retitled to {{branch-4.3}} to avoid confusion with the {{branch-4.x}} runs.
> h3. Dependency
> The Maven workflows inherited by {{branch-4.3}} carry the bug tracked separately: they call {{maven_test.yml}} without passing {{branch:}}, and that workflow's default is {{master}}, so they build {{master}} instead of the branch. That should be fixed first (or at the same time), otherwise {{branch-4.3}}'s new scheduler will faithfully dispatch three Maven workflows that silently test the wrong branch.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org