You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2026/05/14 11:08:47 UTC
[PR] [INFRA] Pin downstream actions/checkout to a single resolved SHA [spark]
zhengruifeng opened a new pull request, #55879:
URL: https://github.com/apache/spark/pull/55879
### What changes were proposed in this pull request?
In `.github/workflows/build_and_test.yml`, add a step to the `precondition` job that captures `git rev-parse HEAD` right after the apache/spark checkout, exposes it as a `head_sha` output, and switch every downstream `actions/checkout` from `ref: ${{ inputs.branch }}` to `ref: ${{ needs.precondition.outputs.head_sha }}`. The `precondition` job's own checkout still resolves `inputs.branch`; the 11 downstream checkouts (`build`, `infra-image`, `precompile`, `pyspark`, `sparkr`, `buf`, `lint`, `docs`, `tpcds-1g`, `docker-integration-tests`, `k8s-integration-tests`) now all pin to the same SHA.
### Why are the changes needed?
Today each `actions/checkout` step independently re-resolves `ref: ${{ inputs.branch }}` (default `master`) at the moment the runner picks it up. Different jobs in the same workflow run can therefore end up testing different commits.
This bites hardest on the `pyspark` matrix because those jobs `needs: [precondition, infra-image, precompile]` and typically start ~17 minutes after the run is created (precompile takes that long). If a commit lands on master in that window, the `pyspark` job downloads a precompiled JAR built from the older commit but checks out Python sources from the newer commit. When the intervening commit adds a tightly-coupled change — new Spark Connect relation + new proto field + new server planner + new Python tests — every NEAREST-BY-style test then fails with:
```
[CONNECT_INVALID_PLAN.INVALID_ONE_OF_FIELD_NOT_SET]
The Spark Connect plan is invalid. This oneOf field in spark.connect.Relation is not set: RELTYPE_NOT_SET
```
Concrete example from 2026-05-14:
- Run [25835824862](https://github.com/apache/spark/actions/runs/25835824862) triggered by `e19bc35c` (SPARK-56844) — `pyspark-connect` failed with 19 NEAREST BY errors.
- Run [25835929554](https://github.com/apache/spark/actions/runs/25835929554) triggered ~3 minutes later by the next commit `13380e78` (SPARK-56395, which added the NEAREST BY feature) — same job passed.
The first run's `precompile` checked out `e19bc35c` (no NEAREST BY server code), but by the time its `pyspark-connect` job actually started, master was at `13380e78` and `actions/checkout` resolved that newer commit (with the new Python test files). Pinning every job to the SHA `precondition` saw makes this impossible.
### Does this PR introduce _any_ user-facing change?
No. CI infrastructure only.
### How was this patch tested?
YAML syntax validated locally. CI will exercise the change end-to-end.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-4-7)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56866][INFRA] Pin downstream actions/checkout to a single resolved SHA [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #55879:
URL: https://github.com/apache/spark/pull/55879#issuecomment-4483539762
thanks, merged into master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56866][INFRA] Pin downstream actions/checkout to a single resolved SHA [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #55879: [SPARK-56866][INFRA] Pin downstream actions/checkout to a single resolved SHA
URL: https://github.com/apache/spark/pull/55879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56866][INFRA] Pin downstream actions/checkout to a single resolved SHA [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #55879:
URL: https://github.com/apache/spark/pull/55879#issuecomment-4483539829
thanks, merged into master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org