You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "sunchao (via GitHub)" <gi...@apache.org> on 2026/09/10 18:09:11 UTC
[PR] ci: share one Linux native build across CI workflows [datafusion-comet]
sunchao opened a new pull request, #5841:
URL: https://github.com/apache/datafusion-comet/pull/5841
## Which issue does this PR close?
Part of #5830. Follow-up to #3249, which shared native artifacts within individual workflows.
## Rationale for this change
The umbrella CI workflow still compiles the same default Linux native library separately for Linux, each selected Spark version, and each selected Iceberg version. A source-change PR normally builds four copies; a full main-branch run builds nine. This change builds one copy and shares it with those consumers, reducing duplicate compilation and runner usage. Actual time savings depend on cache state and will need measurement in CI.
## What changes are included in this PR?
- Add one reusable Linux native producer using the existing Cargo `ci` profile, JDK 17, compiler flags, and main-only cache save policy.
- Make Linux, Spark SQL, and Iceberg callers wait for that producer and pass its artifact through a required input. Keep Spark/JDK-specific compiled JVM artifacts separate.
- Preserve existing path, event, and opt-in label selection. The producer runs whenever any consumer is selected, including Spark-only changes.
- Remove duplicate native compilation and uploads from consumers. Keep Iceberg shard preparation as a small separate job.
- Add configuration guards and regression coverage for producer selection, caller dependencies, artifact mapping, and accidental duplicate producers.
The Linux reusable workflow, including lint and Rust debug tests, now starts after the shared native build. Rust formatting also runs before native compilation. macOS, Rust debug tests, and feature-specific workflows retain their separate builds. Artifact retention remains one day.
## How are these changes tested?
Passed locally:
- `actionlint -color -shellcheck=`
- `python3 dev/ci/check-ci-config.py`
- `python3 dev/ci/test-ci-config.py` (17 tests)
- `python3 dev/ci/test-native-build-selection.py` (10 tests, including all 512 change-flag combinations across event/label cases)
- `python3 dev/ci/check-suites.py`
- `python3 dev/ci/test-iceberg-shards.py` (15 tests) and shard-matrix generation
- Prettier check on the changed Markdown, Apache RAT, and `git diff --check`
This is a workflow-only change; no full native or Spark build was run locally. Hosted CI still needs to validate artifact transfers and consumer execution, including opt-in Spark 3.4/JDK 11. Failed-job reruns can reuse the successful producer's artifact while it is retained; after expiry, rerun the full workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: share one Linux native build across CI workflows [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5841:
URL: https://github.com/apache/datafusion-comet/pull/5841#issuecomment-5691848650
@andygrove, all items in your latest review are addressed in `3c31e5aa8`. I checked the current head against the review and reran the configuration and routing tests.
- Both Linux callers pass the same `cache-refresh-only` and `profiles` inputs. In cache-refresh mode, the four moved Java/Scala lint, compile-only, and Celeborn jobs are skipped, while Rust formatting and the Rust debug cache writer remain active. The Linux JVM test matrix is also guarded. The expressions account for the newer all-profile/nightly route.
- `CACHE_REFRESH_JOBS` now maps each of the three workflows to its retained jobs and reasons. Each Linux caller's inputs are checked independently. Mutation tests cover a dropped guard in the checks workflow, missing/miswired inputs on either caller, and removing or guarding off the cache writers and their prerequisites.
- The contributor CI guide now explains cache ownership across the three files and where new jobs need guards. The `build_linux` policy comment explicitly explains why push must select all three workflows. The earlier Maven bootstrap, Hive routing, JDK 17, and compile-job documentation fixes remain present.
The configuration checker, all 41 configuration tests, and all 23 selection tests pass. Both [commit CI](https://github.com/apache/datafusion-comet/actions/runs/35040404817) and [Spark 4.1 opt-in CI](https://github.com/apache/datafusion-comet/actions/runs/35040413280) passed on this exact head. Together they exercised all nine native consumers, including every Spark 4.1 Hive shard. Hosted push/nightly execution remains separate from this PR coverage; their selection and guards are covered locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: share one Linux native build across CI workflows [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5841:
URL: https://github.com/apache/datafusion-comet/pull/5841#issuecomment-5672837471
@andygrove, both items from your latest review are addressed. Rebased onto `main` at `39da85081`; the new head is `a126925b7` and GitHub reports no merge conflicts.
1. **JDK 17:** carried the Spark 3.4/JDK 17 lint matrix entry and unconditional `JAVA_TOOL_OPTIONS` into `pr_build_linux_checks.yml`. The preflight, Spark 3.4, Iceberg 1.8, and Linux test updates from #5897 are also preserved. All six moved jobs match the current upstream jobs.
2. **Contributor guide:** the compile-only instructions now point to `pr_build_linux_checks.yml`. The suite-matrix instructions still point to `pr_build_linux.yml`, where `linux-test` remains.
The rebase also preserves Spark 3.4's new label/manual-only policy. Updated the selection regression to verify that queue runs select the other eight consumers, that a Spark 3.4-only queue change does not start native compilation, and that labels cannot opt Spark 3.4 back into a queue run. Its label and manual paths remain covered through both Python and the CLI.
Local checks passed: 36 configuration tests, 21 selection tests (including 299,008 combinations), actionlint, suite/benchmark checks, 15 Iceberg shard tests, 4 PR-label tests, Markdown formatting, and whitespace checks. Updated the PR description with the previous green coverage and the [fresh CI run](https://github.com/apache/datafusion-comet/actions/runs/34913499387). The four opt-in labels remain applied; results on this rebased head are pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: share one Linux native build across CI workflows [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5841:
URL: https://github.com/apache/datafusion-comet/pull/5841#issuecomment-5876467963
This is a light fully automated review since there are so many PRs open.
I found three comments outside the changed files that still describe the pre-split layout, so they go stale once this lands. `.scalafix-syntactic.conf:20` sends readers to the `scalafix-syntactic` job in `pr_build_linux.yml`, and `.github/workflows/pr_benchmark_check.yml:75` points at `pr_build_linux.yml` for the same semanticdb exclusion in the main lint matrix. Both `scalafix-syntactic` and `lint-java` now live in `pr_build_linux_checks.yml`. `dev/ci/linux-test-profiles.py:26` says the `lint` job runs the script and publishes the matrix that `linux-test` reads. At this head that is `prepare-matrix` in `pr_build_linux.yml`, and `lint` only runs `cargo fmt` in the checks workflow. Could these three pointers be updated in this PR, so the next person chasing a syntactic scalafix failure or editing the profile rows lands in the right file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: share one Linux native build across CI workflows [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5841:
URL: https://github.com/apache/datafusion-comet/pull/5841#discussion_r4178648990
##########
.github/workflows/spark_sql_test_reusable.yml:
##########
@@ -91,55 +87,17 @@ jobs:
id: modules
run: python3 dev/ci/spark-sql-modules.py --modules "${{ inputs.modules }}" --github-output "$GITHUB_OUTPUT"
- - name: Setup Rust & Java toolchain
- uses: ./.github/actions/setup-builder
- with:
- rust-version: ${{ env.RUST_VERSION }}
- jdk-version: ${{ inputs.java }}
-
- - name: Restore Cargo cache
- uses: actions/cache/restore@v6
+ - name: Setup Java toolchain
+ uses: actions/setup-java@v4
Review Comment:
Addressed in eae558b after rebasing on 98662215d. The Java-only Spark build now calls maven-bootstrap immediately after setup-java. Moved Linux jobs retain main’s setup-builder bootstrap, and the guard recognizes that composite rather than requiring redundant explicit steps. The mutation test now covers loss, lateness, conditional execution and swallowed failures for both bootstrap forms, including the Spark build. All 42 configuration tests, 24 selection tests and actionlint pass. Fresh hosted CI is pending.
##########
dev/ci/compute-changes.py:
##########
@@ -546,11 +577,26 @@ def event_allows(job, event):
def compute(files, event):
- """Return {job: bool}, folding the path filter and the event policy."""
- return {
- name: event_allows(name, event) and matches(patterns, files)
+ """Return a new {output: bool} mapping for consumers and their native build.
+
+ `files` is a reusable sequence of repository-relative changed paths;
+ `event` has the fields described by event_allows(). Neither input is
+ mutated. Manual dispatch selects every route even with no changed files;
+ other events require both path and event matches. The shared native build
+ is selected only after those decisions, taking the union of every caller's
+ routes, so Hive alone can start its producer and a denied opt-in route
+ cannot start an unused producer. Unknown events select nothing. Configuration
+ lookup failures propagate as KeyError rather than returning partial output.
+ """
+ manual = event.get("name") == "workflow_dispatch"
+ outputs = {
+ name: event_allows(name, event) and (manual or matches(patterns, files))
for name, patterns in FILTERS.items()
}
+ outputs["build_linux_native"] = any(
Review Comment:
Added coverage in eae558b for native source/manifest/lockfile/protobuf/config inputs on push through the real selector and CLI. A separate mutation regression adds each contrib manifest to build_linux’s filter and verifies the producer union follows it. Updated #5976 now uses the shared-filter extension and no post-compute override, so its routing decision precedes this union. This branch is rebased on current main; #5976 remains unmerged. The description explicitly preserves the dependency and calls out moving the shared producer to build-native-ci when rebasing onto that dependency before merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: share one Linux native build across CI workflows [datafusion-comet]
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #5841:
URL: https://github.com/apache/datafusion-comet/pull/5841#discussion_r3982217446
##########
.github/workflows/pr_build_linux_checks.yml:
##########
@@ -0,0 +1,262 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: PR Checks (Linux)
+
+# Independent lint, compilation, compatibility, and Rust debug checks.
+# Run alongside the shared native producer; these jobs do not consume its artifact.
+# Triggering and path filters live in the umbrella workflow.
+on:
+ workflow_call:
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+ # Force GNU ld on Linux: recent Rust stable defaults to rust-lld on
+ # x86_64-unknown-linux-gnu, and rust-lld cannot resolve -ljvm against the
+ # Zulu JDK layout installed by setup-java. Keep bfd for all cargo invocations.
+ RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
+
+jobs:
+
+ # Rust formatting gates the build and test jobs below
+ lint:
+ name: Lint
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Check Rust formatting
+ run: |
+ rustup component add rustfmt
+ cd native && cargo fmt --all -- --check
+
+ # Fast syntactic-only scalafix check. Parses sources without compiling, so it
+ # surfaces version-independent style issues (e.g. RedundantSyntax) in seconds,
+ # long before the lint-java matrix finishes its ~3.5 min build. It also scans
+ # the spark-4.1 / spark-4.2 sources that lint-java skips (those profiles can't
+ # run -Psemanticdb yet), so it is the only gate covering them. The full rule
+ # set, including the semantic rules, still runs in lint-java.
+ scalafix-syntactic:
+ name: Lint Scala (syntactic)
+ runs-on: ubuntu-24.04
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup coursier
+ uses: coursier/setup-action@v3
+ with:
+ jvm: temurin:21
+
+ - name: Run syntactic scalafix check (no compile)
+ run: |
+ cs launch scalafix:0.14.6 -- \
+ --check \
+ --syntactic \
+ --config .scalafix-syntactic.conf \
+ --exclude '**/target/**' \
+ spark
+
+ lint-java:
Review Comment:
## CodeQL / Workflow does not contain permissions
Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}}
[Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/150)
##########
.github/workflows/ci.yml:
##########
@@ -179,16 +182,112 @@
# each time somebody applies a label.
# ---------------------------------------------------------------------------
+ # Independent checks start immediately after change selection.
+ pr_build_linux_checks:
+ name: PR Checks (Linux)
+ needs: changes
+ if: |
+ needs.changes.outputs.build_linux == 'true' &&
+ (github.event_name == 'push' ||
+ github.event_name == 'workflow_dispatch' ||
+ (github.event_name == 'pull_request' &&
+ github.event.action != 'labeled'))
+ uses: ./.github/workflows/pr_build_linux_checks.yml
+
+ # Build once for the union of the native-library consumers selected below.
+ # Preserve their event/label gates: a legacy-version diff without its opt-in
+ # label, or an unrelated label event, must not start an unused native build.
+ build_linux_native:
Review Comment:
## CodeQL / Workflow does not contain permissions
Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{}}
[Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/148)
##########
.github/workflows/pr_build_linux_checks.yml:
##########
@@ -0,0 +1,262 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: PR Checks (Linux)
+
+# Independent lint, compilation, compatibility, and Rust debug checks.
+# Run alongside the shared native producer; these jobs do not consume its artifact.
+# Triggering and path filters live in the umbrella workflow.
+on:
+ workflow_call:
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+ # Force GNU ld on Linux: recent Rust stable defaults to rust-lld on
+ # x86_64-unknown-linux-gnu, and rust-lld cannot resolve -ljvm against the
+ # Zulu JDK layout installed by setup-java. Keep bfd for all cargo invocations.
+ RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
+
+jobs:
+
+ # Rust formatting gates the build and test jobs below
+ lint:
+ name: Lint
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Check Rust formatting
+ run: |
+ rustup component add rustfmt
+ cd native && cargo fmt --all -- --check
+
+ # Fast syntactic-only scalafix check. Parses sources without compiling, so it
+ # surfaces version-independent style issues (e.g. RedundantSyntax) in seconds,
+ # long before the lint-java matrix finishes its ~3.5 min build. It also scans
+ # the spark-4.1 / spark-4.2 sources that lint-java skips (those profiles can't
+ # run -Psemanticdb yet), so it is the only gate covering them. The full rule
+ # set, including the semantic rules, still runs in lint-java.
+ scalafix-syntactic:
+ name: Lint Scala (syntactic)
+ runs-on: ubuntu-24.04
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup coursier
+ uses: coursier/setup-action@v3
+ with:
+ jvm: temurin:21
+
+ - name: Run syntactic scalafix check (no compile)
+ run: |
+ cs launch scalafix:0.14.6 -- \
+ --check \
+ --syntactic \
+ --config .scalafix-syntactic.conf \
+ --exclude '**/target/**' \
+ spark
+
+ lint-java:
+ needs: lint
+ name: Lint Java (${{ matrix.profile.name }})
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ env:
+ JAVA_TOOL_OPTIONS: ${{ (matrix.profile.java_version == '17' || matrix.profile.java_version == '21') && '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED' || '' }}
+ strategy:
+ matrix:
+ profile:
+ - name: "Spark 3.4, JDK 11, Scala 2.12"
+ java_version: "11"
+ maven_opts: "-Pspark-3.4 -Pscala-2.12"
+ - name: "Spark 3.5, JDK 17, Scala 2.12"
+ java_version: "17"
+ maven_opts: "-Pspark-3.5 -Pscala-2.12"
+ - name: "Spark 4.0, JDK 17"
+ java_version: "17"
+ maven_opts: "-Pspark-4.0"
+ - name: "Spark 4.0, JDK 21"
+ java_version: "21"
+ maven_opts: "-Pspark-4.0"
+ # Spark 4.1 and 4.2 are intentionally absent: the lint job invokes -Psemanticdb,
+ # but semanticdb-scalac for those Scala patch versions (2.13.17 / 2.13.18) is not
+ # yet published, so we cannot currently run scalafix against the spark-4.1 or
+ # spark-4.2 profiles.
+ fail-fast: false
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: ${{ matrix.profile.java_version }}
+
+ - name: Cache Maven dependencies
+ uses: actions/cache@v6
+ with:
+ path: |
+ ~/.m2/repository
+ /root/.m2/repository
+ key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-lint
+ restore-keys: |
+ ${{ runner.os }}-java-maven-
+
+ - name: Run scalafix check
+ run: |
+ ./mvnw -B package -DskipTests scalafix:scalafix -Dscalafix.mode=CHECK -Psemanticdb ${{ matrix.profile.maven_opts }}
+
+ - name: Setup Node.js
+ uses: actions/setup-node@v7
+ with:
+ node-version: '24'
+
+ - name: Install prettier
+ run: |
+ npm install -g prettier
+
+ - name: Run prettier
+ run: |
+ npx prettier "**/*.md" --write
+
+ - name: Mark workspace as safe for git
+ run: |
+ git config --global --add safe.directory "$GITHUB_WORKSPACE"
+
+ - name: Check for any local git changes (such as generated docs)
+ run: |
+ ./dev/ci/check-working-tree-clean.sh
+
+ # Compile-only verification for Spark 4.1. Tests are intentionally skipped: the spark-4.1
+ # profile is currently a build target only, and several runtime/test failures are tracked
+ # in follow-up PRs. Excluded from lint-java because semanticdb-scalac_2.13.17 is not yet
+ # published and the lint job activates -Psemanticdb.
+ build-spark-4-1:
+ needs: lint
+ name: Build Spark 4.1, JDK 17
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: 17
+
+ - name: Cache Maven dependencies
+ uses: actions/cache@v6
+ with:
+ path: |
+ ~/.m2/repository
+ /root/.m2/repository
+ key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-spark-4.1-build
+ restore-keys: |
+ ${{ runner.os }}-java-maven-
+
+ - name: Compile (skip tests)
+ run: ./mvnw -B install -DskipTests -Dmaven.test.skip=true -Pspark-4.1
+
+ celeborn-reflection-compatibility:
+ needs: lint
+ name: Celeborn ${{ matrix.celeborn_version }} reflection compatibility
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ env:
+ JAVA_TOOL_OPTIONS: --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED
+ strategy:
+ fail-fast: false
+ matrix:
+ celeborn_version: ["0.6.0", "0.7.0"]
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: 17
+
+ - name: Cache Maven dependencies
+ uses: actions/cache@v6
+ with:
+ path: |
+ ~/.m2/repository
+ /root/.m2/repository
+ key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-celeborn-${{ matrix.celeborn_version }}
+ restore-keys: |
+ ${{ runner.os }}-java-maven-
+
+ - name: Verify reflected Celeborn internals
+ env:
+ SPARK_LOCAL_HOSTNAME: localhost
+ SPARK_LOCAL_IP: 127.0.0.1
+ run: |
+ SPARK_HOME="$GITHUB_WORKSPACE" ./mvnw -B clean test \
+ -Pspark-3.5,scala-2.12,celeborn-reflection-compatibility \
+ -Dceleborn.version="${{ matrix.celeborn_version }}" \
+ -Dtest=none \
+ -Dsuites=org.apache.comet.shuffle.CelebornReflectionCompatibilitySuite \
+ -DfailIfNoTests=false
+
+ # Rust tests use a separate debug build, not the shared CI library.
+ linux-test-rust:
Review Comment:
## CodeQL / Workflow does not contain permissions
Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}}
[Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/153)
##########
.github/workflows/pr_build_linux_checks.yml:
##########
@@ -0,0 +1,262 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: PR Checks (Linux)
+
+# Independent lint, compilation, compatibility, and Rust debug checks.
+# Run alongside the shared native producer; these jobs do not consume its artifact.
+# Triggering and path filters live in the umbrella workflow.
+on:
+ workflow_call:
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+ # Force GNU ld on Linux: recent Rust stable defaults to rust-lld on
+ # x86_64-unknown-linux-gnu, and rust-lld cannot resolve -ljvm against the
+ # Zulu JDK layout installed by setup-java. Keep bfd for all cargo invocations.
+ RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
+
+jobs:
+
+ # Rust formatting gates the build and test jobs below
+ lint:
+ name: Lint
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Check Rust formatting
+ run: |
+ rustup component add rustfmt
+ cd native && cargo fmt --all -- --check
+
+ # Fast syntactic-only scalafix check. Parses sources without compiling, so it
+ # surfaces version-independent style issues (e.g. RedundantSyntax) in seconds,
+ # long before the lint-java matrix finishes its ~3.5 min build. It also scans
+ # the spark-4.1 / spark-4.2 sources that lint-java skips (those profiles can't
+ # run -Psemanticdb yet), so it is the only gate covering them. The full rule
+ # set, including the semantic rules, still runs in lint-java.
+ scalafix-syntactic:
+ name: Lint Scala (syntactic)
+ runs-on: ubuntu-24.04
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup coursier
+ uses: coursier/setup-action@v3
+ with:
+ jvm: temurin:21
+
+ - name: Run syntactic scalafix check (no compile)
+ run: |
+ cs launch scalafix:0.14.6 -- \
+ --check \
+ --syntactic \
+ --config .scalafix-syntactic.conf \
+ --exclude '**/target/**' \
+ spark
+
+ lint-java:
+ needs: lint
+ name: Lint Java (${{ matrix.profile.name }})
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ env:
+ JAVA_TOOL_OPTIONS: ${{ (matrix.profile.java_version == '17' || matrix.profile.java_version == '21') && '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED' || '' }}
+ strategy:
+ matrix:
+ profile:
+ - name: "Spark 3.4, JDK 11, Scala 2.12"
+ java_version: "11"
+ maven_opts: "-Pspark-3.4 -Pscala-2.12"
+ - name: "Spark 3.5, JDK 17, Scala 2.12"
+ java_version: "17"
+ maven_opts: "-Pspark-3.5 -Pscala-2.12"
+ - name: "Spark 4.0, JDK 17"
+ java_version: "17"
+ maven_opts: "-Pspark-4.0"
+ - name: "Spark 4.0, JDK 21"
+ java_version: "21"
+ maven_opts: "-Pspark-4.0"
+ # Spark 4.1 and 4.2 are intentionally absent: the lint job invokes -Psemanticdb,
+ # but semanticdb-scalac for those Scala patch versions (2.13.17 / 2.13.18) is not
+ # yet published, so we cannot currently run scalafix against the spark-4.1 or
+ # spark-4.2 profiles.
+ fail-fast: false
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: ${{ matrix.profile.java_version }}
+
+ - name: Cache Maven dependencies
+ uses: actions/cache@v6
+ with:
+ path: |
+ ~/.m2/repository
+ /root/.m2/repository
+ key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-lint
+ restore-keys: |
+ ${{ runner.os }}-java-maven-
+
+ - name: Run scalafix check
+ run: |
+ ./mvnw -B package -DskipTests scalafix:scalafix -Dscalafix.mode=CHECK -Psemanticdb ${{ matrix.profile.maven_opts }}
+
+ - name: Setup Node.js
+ uses: actions/setup-node@v7
+ with:
+ node-version: '24'
+
+ - name: Install prettier
+ run: |
+ npm install -g prettier
+
+ - name: Run prettier
+ run: |
+ npx prettier "**/*.md" --write
+
+ - name: Mark workspace as safe for git
+ run: |
+ git config --global --add safe.directory "$GITHUB_WORKSPACE"
+
+ - name: Check for any local git changes (such as generated docs)
+ run: |
+ ./dev/ci/check-working-tree-clean.sh
+
+ # Compile-only verification for Spark 4.1. Tests are intentionally skipped: the spark-4.1
+ # profile is currently a build target only, and several runtime/test failures are tracked
+ # in follow-up PRs. Excluded from lint-java because semanticdb-scalac_2.13.17 is not yet
+ # published and the lint job activates -Psemanticdb.
+ build-spark-4-1:
+ needs: lint
+ name: Build Spark 4.1, JDK 17
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: 17
+
+ - name: Cache Maven dependencies
+ uses: actions/cache@v6
+ with:
+ path: |
+ ~/.m2/repository
+ /root/.m2/repository
+ key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-spark-4.1-build
+ restore-keys: |
+ ${{ runner.os }}-java-maven-
+
+ - name: Compile (skip tests)
+ run: ./mvnw -B install -DskipTests -Dmaven.test.skip=true -Pspark-4.1
+
+ celeborn-reflection-compatibility:
+ needs: lint
+ name: Celeborn ${{ matrix.celeborn_version }} reflection compatibility
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ env:
+ JAVA_TOOL_OPTIONS: --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED
+ strategy:
+ fail-fast: false
+ matrix:
+ celeborn_version: ["0.6.0", "0.7.0"]
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: 17
+
+ - name: Cache Maven dependencies
+ uses: actions/cache@v6
+ with:
+ path: |
+ ~/.m2/repository
+ /root/.m2/repository
+ key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-celeborn-${{ matrix.celeborn_version }}
+ restore-keys: |
+ ${{ runner.os }}-java-maven-
+
+ - name: Verify reflected Celeborn internals
+ env:
+ SPARK_LOCAL_HOSTNAME: localhost
+ SPARK_LOCAL_IP: 127.0.0.1
+ run: |
+ SPARK_HOME="$GITHUB_WORKSPACE" ./mvnw -B clean test \
+ -Pspark-3.5,scala-2.12,celeborn-reflection-compatibility \
+ -Dceleborn.version="${{ matrix.celeborn_version }}" \
+ -Dtest=none \
+ -Dsuites=org.apache.comet.shuffle.CelebornReflectionCompatibilitySuite \
+ -DfailIfNoTests=false
+
+ # Rust tests use a separate debug build, not the shared CI library.
+ linux-test-rust:
+ needs: lint
+ name: ubuntu-latest/rust-test
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: 17
+
+ - name: Restore Cargo cache
+ uses: actions/cache/restore@v6
+ with:
+ path: |
+ ~/.cargo/registry
+ ~/.cargo/git
+ native/target
+ # Note: Java version intentionally excluded - Rust target is JDK-independent
+ key: ${{ runner.os }}-cargo-debug-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
+ restore-keys: |
+ ${{ runner.os }}-cargo-debug-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-
+
+ - name: Rust test steps
+ uses: ./.github/actions/rust-test
+
+ - name: Save Cargo cache
+ uses: actions/cache/save@v6
+ if: github.ref == 'refs/heads/main'
+ with:
+ path: |
+ ~/.cargo/registry
+ ~/.cargo/git
+ native/target
+ key: ${{ runner.os }}-cargo-debug-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
Review Comment:
## CodeQL / Workflow does not contain permissions
Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}}
[Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/154)
##########
.github/workflows/pr_build_linux_checks.yml:
##########
@@ -0,0 +1,262 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: PR Checks (Linux)
+
+# Independent lint, compilation, compatibility, and Rust debug checks.
+# Run alongside the shared native producer; these jobs do not consume its artifact.
+# Triggering and path filters live in the umbrella workflow.
+on:
+ workflow_call:
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+ # Force GNU ld on Linux: recent Rust stable defaults to rust-lld on
+ # x86_64-unknown-linux-gnu, and rust-lld cannot resolve -ljvm against the
+ # Zulu JDK layout installed by setup-java. Keep bfd for all cargo invocations.
+ RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
+
+jobs:
+
+ # Rust formatting gates the build and test jobs below
+ lint:
+ name: Lint
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Check Rust formatting
+ run: |
+ rustup component add rustfmt
+ cd native && cargo fmt --all -- --check
+
+ # Fast syntactic-only scalafix check. Parses sources without compiling, so it
+ # surfaces version-independent style issues (e.g. RedundantSyntax) in seconds,
+ # long before the lint-java matrix finishes its ~3.5 min build. It also scans
+ # the spark-4.1 / spark-4.2 sources that lint-java skips (those profiles can't
+ # run -Psemanticdb yet), so it is the only gate covering them. The full rule
+ # set, including the semantic rules, still runs in lint-java.
+ scalafix-syntactic:
+ name: Lint Scala (syntactic)
+ runs-on: ubuntu-24.04
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup coursier
+ uses: coursier/setup-action@v3
+ with:
+ jvm: temurin:21
+
+ - name: Run syntactic scalafix check (no compile)
+ run: |
+ cs launch scalafix:0.14.6 -- \
+ --check \
+ --syntactic \
+ --config .scalafix-syntactic.conf \
+ --exclude '**/target/**' \
+ spark
+
+ lint-java:
+ needs: lint
+ name: Lint Java (${{ matrix.profile.name }})
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ env:
+ JAVA_TOOL_OPTIONS: ${{ (matrix.profile.java_version == '17' || matrix.profile.java_version == '21') && '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED' || '' }}
+ strategy:
+ matrix:
+ profile:
+ - name: "Spark 3.4, JDK 11, Scala 2.12"
+ java_version: "11"
+ maven_opts: "-Pspark-3.4 -Pscala-2.12"
+ - name: "Spark 3.5, JDK 17, Scala 2.12"
+ java_version: "17"
+ maven_opts: "-Pspark-3.5 -Pscala-2.12"
+ - name: "Spark 4.0, JDK 17"
+ java_version: "17"
+ maven_opts: "-Pspark-4.0"
+ - name: "Spark 4.0, JDK 21"
+ java_version: "21"
+ maven_opts: "-Pspark-4.0"
+ # Spark 4.1 and 4.2 are intentionally absent: the lint job invokes -Psemanticdb,
+ # but semanticdb-scalac for those Scala patch versions (2.13.17 / 2.13.18) is not
+ # yet published, so we cannot currently run scalafix against the spark-4.1 or
+ # spark-4.2 profiles.
+ fail-fast: false
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: ${{ matrix.profile.java_version }}
+
+ - name: Cache Maven dependencies
+ uses: actions/cache@v6
+ with:
+ path: |
+ ~/.m2/repository
+ /root/.m2/repository
+ key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-lint
+ restore-keys: |
+ ${{ runner.os }}-java-maven-
+
+ - name: Run scalafix check
+ run: |
+ ./mvnw -B package -DskipTests scalafix:scalafix -Dscalafix.mode=CHECK -Psemanticdb ${{ matrix.profile.maven_opts }}
+
+ - name: Setup Node.js
+ uses: actions/setup-node@v7
+ with:
+ node-version: '24'
+
+ - name: Install prettier
+ run: |
+ npm install -g prettier
+
+ - name: Run prettier
+ run: |
+ npx prettier "**/*.md" --write
+
+ - name: Mark workspace as safe for git
+ run: |
+ git config --global --add safe.directory "$GITHUB_WORKSPACE"
+
+ - name: Check for any local git changes (such as generated docs)
+ run: |
+ ./dev/ci/check-working-tree-clean.sh
+
+ # Compile-only verification for Spark 4.1. Tests are intentionally skipped: the spark-4.1
+ # profile is currently a build target only, and several runtime/test failures are tracked
+ # in follow-up PRs. Excluded from lint-java because semanticdb-scalac_2.13.17 is not yet
+ # published and the lint job activates -Psemanticdb.
+ build-spark-4-1:
Review Comment:
## CodeQL / Workflow does not contain permissions
Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}}
[Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/151)
##########
.github/workflows/pr_build_linux_checks.yml:
##########
@@ -0,0 +1,262 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: PR Checks (Linux)
+
+# Independent lint, compilation, compatibility, and Rust debug checks.
+# Run alongside the shared native producer; these jobs do not consume its artifact.
+# Triggering and path filters live in the umbrella workflow.
+on:
+ workflow_call:
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+ # Force GNU ld on Linux: recent Rust stable defaults to rust-lld on
+ # x86_64-unknown-linux-gnu, and rust-lld cannot resolve -ljvm against the
+ # Zulu JDK layout installed by setup-java. Keep bfd for all cargo invocations.
+ RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
+
+jobs:
+
+ # Rust formatting gates the build and test jobs below
+ lint:
+ name: Lint
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Check Rust formatting
+ run: |
+ rustup component add rustfmt
+ cd native && cargo fmt --all -- --check
+
+ # Fast syntactic-only scalafix check. Parses sources without compiling, so it
+ # surfaces version-independent style issues (e.g. RedundantSyntax) in seconds,
+ # long before the lint-java matrix finishes its ~3.5 min build. It also scans
+ # the spark-4.1 / spark-4.2 sources that lint-java skips (those profiles can't
+ # run -Psemanticdb yet), so it is the only gate covering them. The full rule
+ # set, including the semantic rules, still runs in lint-java.
+ scalafix-syntactic:
+ name: Lint Scala (syntactic)
+ runs-on: ubuntu-24.04
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup coursier
+ uses: coursier/setup-action@v3
+ with:
+ jvm: temurin:21
+
+ - name: Run syntactic scalafix check (no compile)
+ run: |
+ cs launch scalafix:0.14.6 -- \
+ --check \
+ --syntactic \
+ --config .scalafix-syntactic.conf \
+ --exclude '**/target/**' \
+ spark
+
+ lint-java:
+ needs: lint
+ name: Lint Java (${{ matrix.profile.name }})
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ env:
+ JAVA_TOOL_OPTIONS: ${{ (matrix.profile.java_version == '17' || matrix.profile.java_version == '21') && '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED' || '' }}
+ strategy:
+ matrix:
+ profile:
+ - name: "Spark 3.4, JDK 11, Scala 2.12"
+ java_version: "11"
+ maven_opts: "-Pspark-3.4 -Pscala-2.12"
+ - name: "Spark 3.5, JDK 17, Scala 2.12"
+ java_version: "17"
+ maven_opts: "-Pspark-3.5 -Pscala-2.12"
+ - name: "Spark 4.0, JDK 17"
+ java_version: "17"
+ maven_opts: "-Pspark-4.0"
+ - name: "Spark 4.0, JDK 21"
+ java_version: "21"
+ maven_opts: "-Pspark-4.0"
+ # Spark 4.1 and 4.2 are intentionally absent: the lint job invokes -Psemanticdb,
+ # but semanticdb-scalac for those Scala patch versions (2.13.17 / 2.13.18) is not
+ # yet published, so we cannot currently run scalafix against the spark-4.1 or
+ # spark-4.2 profiles.
+ fail-fast: false
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: ${{ matrix.profile.java_version }}
+
+ - name: Cache Maven dependencies
+ uses: actions/cache@v6
+ with:
+ path: |
+ ~/.m2/repository
+ /root/.m2/repository
+ key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-lint
+ restore-keys: |
+ ${{ runner.os }}-java-maven-
+
+ - name: Run scalafix check
+ run: |
+ ./mvnw -B package -DskipTests scalafix:scalafix -Dscalafix.mode=CHECK -Psemanticdb ${{ matrix.profile.maven_opts }}
+
+ - name: Setup Node.js
+ uses: actions/setup-node@v7
+ with:
+ node-version: '24'
+
+ - name: Install prettier
+ run: |
+ npm install -g prettier
+
+ - name: Run prettier
+ run: |
+ npx prettier "**/*.md" --write
+
+ - name: Mark workspace as safe for git
+ run: |
+ git config --global --add safe.directory "$GITHUB_WORKSPACE"
+
+ - name: Check for any local git changes (such as generated docs)
+ run: |
+ ./dev/ci/check-working-tree-clean.sh
+
+ # Compile-only verification for Spark 4.1. Tests are intentionally skipped: the spark-4.1
+ # profile is currently a build target only, and several runtime/test failures are tracked
+ # in follow-up PRs. Excluded from lint-java because semanticdb-scalac_2.13.17 is not yet
+ # published and the lint job activates -Psemanticdb.
+ build-spark-4-1:
+ needs: lint
+ name: Build Spark 4.1, JDK 17
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: 17
+
+ - name: Cache Maven dependencies
+ uses: actions/cache@v6
+ with:
+ path: |
+ ~/.m2/repository
+ /root/.m2/repository
+ key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-spark-4.1-build
+ restore-keys: |
+ ${{ runner.os }}-java-maven-
+
+ - name: Compile (skip tests)
+ run: ./mvnw -B install -DskipTests -Dmaven.test.skip=true -Pspark-4.1
+
+ celeborn-reflection-compatibility:
Review Comment:
## CodeQL / Workflow does not contain permissions
Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}}
[Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/152)
##########
.github/workflows/pr_build_linux_checks.yml:
##########
@@ -0,0 +1,262 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: PR Checks (Linux)
+
+# Independent lint, compilation, compatibility, and Rust debug checks.
+# Run alongside the shared native producer; these jobs do not consume its artifact.
+# Triggering and path filters live in the umbrella workflow.
+on:
+ workflow_call:
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+ # Force GNU ld on Linux: recent Rust stable defaults to rust-lld on
+ # x86_64-unknown-linux-gnu, and rust-lld cannot resolve -ljvm against the
+ # Zulu JDK layout installed by setup-java. Keep bfd for all cargo invocations.
+ RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
+
+jobs:
+
+ # Rust formatting gates the build and test jobs below
+ lint:
+ name: Lint
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Check Rust formatting
+ run: |
+ rustup component add rustfmt
+ cd native && cargo fmt --all -- --check
+
+ # Fast syntactic-only scalafix check. Parses sources without compiling, so it
+ # surfaces version-independent style issues (e.g. RedundantSyntax) in seconds,
+ # long before the lint-java matrix finishes its ~3.5 min build. It also scans
+ # the spark-4.1 / spark-4.2 sources that lint-java skips (those profiles can't
+ # run -Psemanticdb yet), so it is the only gate covering them. The full rule
+ # set, including the semantic rules, still runs in lint-java.
+ scalafix-syntactic:
Review Comment:
## CodeQL / Workflow does not contain permissions
Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}}
[Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/149)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: share one Linux native build across CI workflows [datafusion-comet]
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #5841:
URL: https://github.com/apache/datafusion-comet/pull/5841#discussion_r3982133363
##########
.github/workflows/build_linux_native.yml:
##########
@@ -0,0 +1,82 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Single producer for the default Linux CI native library. The umbrella runs
+# this whenever any Linux, Spark SQL, or Iceberg test consumer is selected.
+name: Build Linux Native Library
+
+on:
+ workflow_call:
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+
+jobs:
+ # Build native library once and share with all test jobs
+ build-native:
+ name: Build Native Library
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ steps:
+ - uses: actions/checkout@v7
+ - name: Setup Rust toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: 17 # Matches the existing Linux producer, including libjvm linkage.
+
+ - name: Check Rust formatting
+ run: cd native && cargo fmt --all -- --check
+
+ - name: Restore Cargo cache
+ uses: actions/cache/restore@v6
+ with:
+ path: |
+ ~/.cargo/registry
+ ~/.cargo/git
+ native/target
+ key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
+ restore-keys: |
+ ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-
+
+ - name: Build native library (CI profile)
+ run: |
+ cd native
+ # CI profile: same overflow behavior as release, but faster compilation
+ # (no LTO, parallel codegen)
+ cargo build --profile ci
+ env:
+ RUSTFLAGS: "-Ctarget-cpu=x86-64-v3 -Clink-arg=-fuse-ld=bfd"
+
+ - name: Upload native library
+ uses: ./.github/actions/upload-artifact-retry
+ with:
+ name: native-lib-linux
+ path: native/target/ci/libcomet.so
+ retention-days: 1
+
+ - name: Save Cargo cache
+ uses: actions/cache/save@v6
+ if: github.ref == 'refs/heads/main'
+ with:
+ path: |
+ ~/.cargo/registry
+ ~/.cargo/git
+ native/target
+ key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
Review Comment:
## CodeQL / Workflow does not contain permissions
Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}}
[Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/137)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: share one Linux native build across CI workflows [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5841:
URL: https://github.com/apache/datafusion-comet/pull/5841#discussion_r3989409782
##########
.github/workflows/ci.yml:
##########
@@ -184,11 +187,41 @@ jobs:
# can actually test it.
# ---------------------------------------------------------------------------
+ # Independent checks start immediately after change selection.
+ pr_build_linux_checks:
+ name: PR Checks (Linux)
+ needs: changes
+ permissions:
+ contents: read
+ if: needs.changes.outputs.build_linux == 'true'
+ uses: ./.github/workflows/pr_build_linux_checks.yml
+
+ # Build once when any native-library consumer is selected. Each output
+ # already includes the path, event, and label policy from compute-changes.py.
+ build_linux_native:
+ name: Shared Linux Native Library
+ needs: changes
+ permissions:
+ contents: read
+ if: |
Review Comment:
#5850 moved routing policy out of `ci.yml` expressions and into `compute-changes.py`, and this nine-way OR is the same kind of policy. Would it work to compute a `build_linux_native` output there as the union of the consumer outputs, so this becomes `needs.changes.outputs.build_linux_native == 'true'`? Then `test-native-build-selection.py` would not need to regex-parse the YAML `if:` block to prove the two sides agree, and the README routing table gets a single row like the others.
##########
.github/workflows/spark_sql_test_reusable.yml:
##########
@@ -52,22 +56,11 @@ env:
jobs:
- # Build the native library AND pre-compile Spark sources + Test classes in a
- # single runner, then publish two artifacts the matrix consumes:
- # - native-lib-spark-<full>-jdk<N>: libcomet.so (~50 MB)
- # - jvm-compiled-spark-<full>-jdk<N>: apache-spark.tar.gz (sources +
- # target/ + Zinc state, ~500 MB-1 GB)
- # Combining them avoids a second runner cold-start and an extra inter-job
- # artifact round-trip for the native lib, since the JVM build already
- # depends on it (the Comet Maven install bundles libcomet.so into the
- # Comet JAR before SBT resolves Spark's classpath).
- #
- # Both names carry the Spark/JDK version because ci.yml calls this workflow
- # once per Spark version inside a single run, and artifact names are scoped
- # to the run, not to the calling workflow. See "Artifact names must be unique
- # per producer" in .github/workflows/README.md.
+ # Pre-compile Spark sources and test classes once per Spark/JDK version.
+ # The native library comes from the umbrella's shared Linux producer; only
+ # the version-specific JVM artifact is published by this workflow.
build:
- name: Build Native + JVM Test Classes
+ name: Build JVM Test Classes
runs-on: ubuntu-24.04
container:
image: amd64/rust
Review Comment:
Now that this job only compiles JVM test classes, does it still need the `amd64/rust` container and the Rust toolchain from `setup-builder`? If the skip-native-build Maven path genuinely needs them, a short comment here would save the next reader the same question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: share one Linux native build across CI workflows [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5841:
URL: https://github.com/apache/datafusion-comet/pull/5841#issuecomment-5635082166
One more heads-up from the merge-queue side, following on from my review above.
#5843 (the PR that actually enables the queue, stacked on #5842) changes `POLICY` in `dev/ci/compute-changes.py` so that a `push` to `main` selects only `docs`. Every test job moves to the `merge_group` event instead, because the queue has already tested the exact tree that lands.
That will break `test_main_and_manual_runs_include_legacy_consumers` in `dev/ci/test-native-build-selection.py` here, which asserts `event="push"` selects every native consumer. Whichever of #5841 and #5843 lands second needs to update that test (the `workflow_dispatch` half stays true). If this PR lands first, #5843 will pick it up on its rebase along with adding `pr_build_linux_checks` and `build_linux_native` to `required_checks.needs`. If #5843 lands first, the `push` assertion here should become `merge_group`.
Also FYI, `run-spark-3.4-tests` and `run-spark-3.5-tests` now exist as labels, so the label-gated selection tests here have real labels behind them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: share one Linux native build across CI workflows [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5841:
URL: https://github.com/apache/datafusion-comet/pull/5841#discussion_r4000818215
##########
.github/workflows/ci.yml:
##########
@@ -184,11 +187,41 @@ jobs:
# can actually test it.
# ---------------------------------------------------------------------------
+ # Independent checks start immediately after change selection.
+ pr_build_linux_checks:
+ name: PR Checks (Linux)
+ needs: changes
+ permissions:
+ contents: read
+ if: needs.changes.outputs.build_linux == 'true'
+ uses: ./.github/workflows/pr_build_linux_checks.yml
+
+ # Build once when any native-library consumer is selected. Each output
+ # already includes the path, event, and label policy from compute-changes.py.
+ build_linux_native:
+ name: Shared Linux Native Library
+ needs: changes
+ permissions:
+ contents: read
+ if: |
Review Comment:
Addressed: `compute-changes.py` now derives `build_linux_native` from the selected consumer routes, and `ci.yml` reads that one output. The mapping also supports Spark 4.1's core and Hive routes selecting the same caller, including Hive-only label events. The selection tests exercise Python and the real CLI; the configuration guard separately verifies exact caller gates and output exports. Updated in 79c161b293f2147b1d25233a47a15222d7289cf9; 21 selection tests pass, including 299,008 path/event combinations.
##########
.github/workflows/spark_sql_test_reusable.yml:
##########
@@ -52,22 +56,11 @@ env:
jobs:
- # Build the native library AND pre-compile Spark sources + Test classes in a
- # single runner, then publish two artifacts the matrix consumes:
- # - native-lib-spark-<full>-jdk<N>: libcomet.so (~50 MB)
- # - jvm-compiled-spark-<full>-jdk<N>: apache-spark.tar.gz (sources +
- # target/ + Zinc state, ~500 MB-1 GB)
- # Combining them avoids a second runner cold-start and an extra inter-job
- # artifact round-trip for the native lib, since the JVM build already
- # depends on it (the Comet Maven install bundles libcomet.so into the
- # Comet JAR before SBT resolves Spark's classpath).
- #
- # Both names carry the Spark/JDK version because ci.yml calls this workflow
- # once per Spark version inside a single run, and artifact names are scoped
- # to the run, not to the calling workflow. See "Artifact names must be unique
- # per producer" in .github/workflows/README.md.
+ # Pre-compile Spark sources and test classes once per Spark/JDK version.
+ # The native library comes from the umbrella's shared Linux producer; only
+ # the version-specific JVM artifact is published by this workflow.
build:
- name: Build Native + JVM Test Classes
+ name: Build JVM Test Classes
runs-on: ubuntu-24.04
container:
image: amd64/rust
Review Comment:
Removed the Rust toolchain setup from the JVM compilation job: it now uses `actions/setup-java@v4`. The adjacent comment explains why the container remains: Spark's precompiled JVM classes and Maven cache are reused by downstream test containers, so their workspace and cache paths must stay consistent. The Spark 3.4/JDK 11 opt-in run has now been requested on rebased head 79c161b293f2147b1d25233a47a15222d7289cf9 to cover this setup path; its result is still pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: share one Linux native build across CI workflows [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5841:
URL: https://github.com/apache/datafusion-comet/pull/5841#discussion_r4112539608
##########
.github/workflows/spark_sql_test_reusable.yml:
##########
@@ -91,55 +87,17 @@ jobs:
id: modules
run: python3 dev/ci/spark-sql-modules.py --modules "${{ inputs.modules }}" --github-output "$GITHUB_OUTPUT"
- - name: Setup Rust & Java toolchain
- uses: ./.github/actions/setup-builder
- with:
- rust-version: ${{ env.RUST_VERSION }}
- jdk-version: ${{ inputs.java }}
-
- - name: Restore Cargo cache
- uses: actions/cache/restore@v6
+ - name: Setup Java toolchain
+ uses: actions/setup-java@v4
Review Comment:
Since #5881 the retrying Maven bootstrap runs inside `setup-builder`, so once this is rebased, swapping `setup-builder` for a bare `setup-java` here leaves `Build JVM Test Classes` as the only job that runs `./mvnw` (through `setup-spark-builder`) without it. `linux_maven_bootstrap_failures` exempts composite actions and main only has routing cases for the bootstrap, so nothing would fail until a Maven Central reset takes out the queue's Spark SQL build. Could this job call `./.github/actions/maven-bootstrap` right after `setup-java`? The explicit `Bootstrap Maven` steps in `pr_build_linux_checks.yml` also become redundant with `setup-builder` after the rebase, and the guard would need to accept `setup-builder` as a bootstrap if those steps go.
##########
dev/ci/compute-changes.py:
##########
@@ -546,11 +577,26 @@ def event_allows(job, event):
def compute(files, event):
- """Return {job: bool}, folding the path filter and the event policy."""
- return {
- name: event_allows(name, event) and matches(patterns, files)
+ """Return a new {output: bool} mapping for consumers and their native build.
+
+ `files` is a reusable sequence of repository-relative changed paths;
+ `event` has the fields described by event_allows(). Neither input is
+ mutated. Manual dispatch selects every route even with no changed files;
+ other events require both path and event matches. The shared native build
+ is selected only after those decisions, taking the union of every caller's
+ routes, so Hive alone can start its producer and a denied opt-in route
+ cannot start an unused producer. Unknown events select nothing. Configuration
+ lookup failures propagate as KeyError rather than returning partial output.
+ """
+ manual = event.get("name") == "workflow_dispatch"
+ outputs = {
+ name: event_allows(name, event) and (manual or matches(patterns, files))
for name, patterns in FILTERS.items()
}
+ outputs["build_linux_native"] = any(
Review Comment:
When this is rebased onto #5976, its push-only override in `compute()`, which sets `build_linux` for `NATIVE_LIBRARY_INPUTS` on push, has to run before this union. If it is appended after it, a push that changes only `contrib/lance/native/Cargo.toml` selects `build_linux` but not `build_linux_native`, so the producer that owns main's native caches never runs for it. I checked that by applying #5976's override after this `compute()` at this head. #5976's `test_native_input_routing` asserts only `build_linux`, so it would stay green either way. Could a test here assert `build_linux_native` for a push of each native-only input? If #5976 takes comphead's suggestion to extend `FILTERS["build_linux"]` instead of overriding, this goes away.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: share one Linux native build across CI workflows [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5841:
URL: https://github.com/apache/datafusion-comet/pull/5841#issuecomment-5655980034
@andygrove, addressed the three items from your latest review and rebased onto `main` at `451c999632`; the PR is now at `79c161b293` with no merge conflicts.
1. **Maven bootstrap:** preserved the bootstrap in all three moved jobs and both TPC jobs, and corrected the routing comment. The new guard requires an earlier unconditional bootstrap with failures propagated before direct `./mvnw` commands in both Linux workflows. Regression mutations remove, move, condition, or ignore the bootstrap's failure in each of the five jobs.
2. **Hive routing:** `NATIVE_CONSUMERS` now maps caller IDs to all selecting outputs. `spark_4_1` includes both core and Hive, so a Hive-only label starts the native producer. Exact caller, OR-gate, export, and dependency validation remains enforced. Compute and CLI tests cover core-only, Hive-only, both, queue, and manual runs; upstream's module selection is preserved.
3. **Coverage:** applied all four requested labels after pushing this head: `run-spark-3.4-tests`, `run-spark-3.5-tests`, `run-spark-4.0-tests`, and `run-iceberg-tests`. The [regular PR run](https://github.com/apache/datafusion-comet/actions/runs/34781345188) and four additional opt-in runs ([1](https://github.com/apache/datafusion-comet/actions/runs/34781354519), [2](https://github.com/apache/datafusion-comet/actions/runs/34781354752), [3](https://github.com/apache/datafusion-comet/actions/runs/34781354819), [4](https://github.com/apache/datafusion-comet/actions/runs/34781355034)) are pending. This requests coverage of all nine rewired consumers, including the Spark 3.4/JDK 11 setup path; runtime results are not yet confirmed.
Local validation passed: 36 configuration regression tests, 21 native-selection tests (299,008 combinations), actionlint, suite/benchmark inventory checks, 15 Iceberg shard tests, 4 PR-label tests, Markdown formatting, and whitespace checks. The PR description now reflects the implementation and pending hosted coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: share one Linux native build across CI workflows [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5841:
URL: https://github.com/apache/datafusion-comet/pull/5841#issuecomment-5982664498
Rebased onto 98662215d and pushed eae558b. The moved jobs come from current main, preserving Spark 4.1/4.2 compilation, strict Scala warnings, main-only cache writes, uncached Rust debug tests, and macOS cache-refresh routing. Spark 4.2 now participates in the shared native producer’s selection and artifact dependency.
Updated the scalafix-syntactic and linux-test-profiles pointers. The old semanticdb pointer in pr_benchmark_check.yml was already removed on main; the rebased file has no stale pointer. Updated the compile-only contributor guidance and PR description, including the remaining #5976 dependency and the absence of current end-to-end timing measurements.
CI configuration, 42 mutation tests, 24 selection tests, actionlint, suite inventory, 15 Iceberg tests, benchmark inventory, Markdown formatting and whitespace checks pass locally. Fresh hosted validation is pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: share one Linux native build across CI workflows [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5841:
URL: https://github.com/apache/datafusion-comet/pull/5841#issuecomment-6003685838
Rebased on main `0ac4dadae7`, bringing in #6646 (`8a7e664`), which fixes the exact Spark 3.4 SPARK-34637 broadcast-reuse failure from the prior head. The exact upstream test now passes locally with Comet enabled, alongside five related DPP/stale-shuffle regressions. All 66 CI configuration/selection tests and actionlint pass; the CI split itself remains intact. Fresh hosted validation is pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org