You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2026/08/04 12:06:00 UTC
[jira] [Assigned] (SPARK-58538) Add branch-4.3 CI scheduler and release integration
[ https://issues.apache.org/jira/browse/SPARK-58538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim reassigned SPARK-58538:
------------------------------------
Assignee: Jungtaek Lim
> Add branch-4.3 CI scheduler and release integration
> ---------------------------------------------------
>
> Key: SPARK-58538
> URL: https://issues.apache.org/jira/browse/SPARK-58538
> Project: Spark
> Issue Type: Improvement
> Components: Project Infra
> Affects Versions: 4.3.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> {{branch-4.3}} was cut from {{branch-4.x}} for the 4.3.0 release, but the CI and release plumbing that {{master}} maintains for each active release branch has not been added for it yet.
> The current model comes from the {{branch-4.2}} work, which happened in two phases: SPARK-58180 (#57319) prepared the build workflows on {{branch-4.2}} so they are self-contained and dispatchable, then SPARK-58242 (#57408) added a single {{branch42_scheduler.yml}} on {{master}} and removed the old per-build {{build_branch42_*.yml}} files from {{master}}. That superseded the earlier SPARK-56710 approach, in which the per-branch build files lived on {{master}}. Only scheduled workflows on the default branch fire, so the scheduler must live on {{master}} while everything it triggers lives on the branch.
> h3. Work items
> On {{master}}:
> * Add {{.github/workflows/branch43_scheduler.yml}}, modelled on {{branch42_scheduler.yml}}: cron entries plus a {{workflow_dispatch}} {{target}} choice, dispatching each build workflow with {{gh workflow run <workflow>.yml --repo ${{ github.repository }} --ref branch-4.3}}. Cron times should be staggered against the existing schedulers to spread runner load.
> * Add {{branch-4.3}} to the branch list in {{.github/workflows/publish_snapshot.yml}} (both the {{workflow_dispatch}} default and the matrix default), which is currently {{'["master", "branch-4.x", "branch-4.2", "branch-4.1", "branch-4.0", "branch-3.5"]'}}.
> * Add a {{branch-4.3}} badge row to {{README.md}}, following the current release-branch form: badges point at the self-contained workflows filtered by {{?branch=branch-4.3}} (including {{build_main.yml}}), as SPARK-58242 reworked the {{branch-4.2}} row to do.
> The eight workflows the scheduler dispatches are {{build_java17}}, {{build_java21}}, {{build_java25}}, {{build_maven}}, {{build_maven_java21}}, {{build_non_ansi}}, {{build_python_3.11}} and {{build_python_3.14}}, matching the {{branch-4.2}} and {{branch-4.x}} schedulers. Note {{build_maven_java25.yml}} exists on the branch but is deliberately not dispatched by any existing scheduler, so it is left out here too.
> On {{branch-4.3}} (the SPARK-58180 equivalent):
> * Remove the dormant {{schedule:}} triggers from the nine inherited {{build_*.yml}} workflows, leaving {{workflow_dispatch:}}. Scheduled runs never fire from a non-default branch, so these crons are inert, and the scheduler on {{master}} is what triggers them. {{branch-4.2}} has {{workflow_dispatch:}} only.
> * Drop the branch token from the workflow titles. The inherited names read e.g. {{name: "Build / Maven (branch-4.x, Scala 2.13, Hadoop 3, JDK 17)"}}, which both collides confusingly with the real {{branch-4.x}} runs in the Actions UI and check-run names, and is the wrong convention: release branches use generic, non-branch-tagged names, e.g. {{branch-4.2}} has {{name: "Build / Maven (Scala 2.13, Hadoop 3, JDK 17)"}}. The branch is implied by where the file lives. So the token should be removed, not changed to {{branch-4.3}}.
> * Point the {{branch}} input default of both reusable workflows at this branch: {{build_and_test.yml}} (inherited as {{branch-4.x}}) and {{maven_test.yml}} (inherited as {{master}}) both to {{branch-4.3}}, so the callers need no explicit {{branch:}}. On {{branch-4.2}} and {{branch-4.1}} both default to their own branch.
> h3. Verified as needing no change
> * {{dev/create-release/release-util.sh}} derives the latest release branch by excluding {{master}} and {{branch-4.x}} and taking the highest remaining. Checked against the real remote: it already resolves to {{branch-4.3}}.
> * {{dev/merge_spark_pr.py}} was generalized by SPARK-56710 to handle any {{branch-4.N}} / {{branch-M.x}}; it hardcodes no branch list and its 70 doctests pass unchanged.
> h3. Dependency
> The {{maven_test.yml}} default on {{branch-4.3}} is the {{branch-4.3}} instance of SPARK-58537: five callers rely on it ({{build_maven}}, {{build_maven_java21}}, {{build_maven_java25}}, {{build_maven_java21_arm}} and {{build_maven_java21_macos26}}), so until it is fixed the Maven workflows build {{master}} rather than the branch. It must be fixed before or with the new scheduler, otherwise the scheduler will faithfully dispatch Maven workflows that silently test the wrong branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org