You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "felipepessoto (via GitHub)" <gi...@apache.org> on 2026/06/28 07:33:41 UTC
[PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
felipepessoto opened a new pull request, #12388:
URL: https://github.com/apache/gluten/pull/12388
Fix https://github.com/apache/gluten/issues/9296.
## What changes are proposed in this pull request?
Adds a CI pipeline that runs delta-io/delta's `spark` ScalaTest suite against the Gluten Velox bundle, so we can validate Gluten against a real Delta release and catch regressions over time.
Running the Delta UTs on Gluten produces **many expected failures** (Gluten does not yet offload every Delta code path, and falls back or behaves differently in places). A plain "red on any failure" gate would be useless. Instead, the pipeline keeps a **committed baseline of known failures** and gates each run against it:
- **regression** -- a test fails that is *not* in the baseline -> the shard fails.
- **expected** -- a failing test that *is* in the baseline -> ignored.
- **now-passing** -- a baseline test that starts passing -> fails the shard (keeps the baseline honest), unless `fail_on_fixed=false`.
### How it works
1. Runs as a **reusable workflow** (`on: workflow_call`) invoked from `velox_backend_x86.yml`, so it **reuses the Velox native libs + Arrow jars that workflow already builds** instead of duplicating the expensive native C++ build. It then assembles the `gluten-velox-bundle` fat jar (Spark 4.1 + Scala 2.13 + JDK 17, Delta profile). A `workflow_dispatch` trigger is kept for standalone manual runs (which build the native lib themselves).
2. Clones delta-io/delta at a release tag (currently `v4.2.0`), drops the bundle onto the `spark` project's test classpath, patches `DeltaSQLCommandTest` to register `GlutenPlugin`, and cherry-picks two merged upstream Delta test-only fixes (delta-io/delta#7104 + #7105) that widen `FileSourceScanExec` checks to `FileSourceScanLike` so Gluten's transformed plan is recognized.
3. Runs `sbt spark/test` **sharded by suite** across **4 shards (4 forked test JVMs each, ~16-way parallelism)**, with ScalaTest's JUnit XML reporter enabled, then gates each shard with `compare-test-results.py` against `known-failures.txt`. A final job aggregates all shards into a single ready-to-commit baseline and flags stale entries.
### Files
| File | Purpose |
|---|---|
| `.github/workflows/velox_backend_x86.yml` | Caller: builds the native lib once, uploads the native + Arrow artifacts, and invokes the reusable Delta workflow (reusing that build instead of duplicating it). |
| `.github/workflows/delta_spark_ut.yml` | The reusable Delta workflow (build bundle -> shard tests -> gate). |
| `.github/workflows/util/delta-spark-ut/setup-delta.sh` | Clones Delta, injects the Gluten bundle, patches `DeltaSQLCommandTest`, cherry-picks the upstream test fixes. |
| `.github/workflows/util/delta-spark-ut/compare-test-results.py` | Parses JUnit XML and enforces / seeds / aggregates against the baseline (stdlib only). |
| `.github/workflows/util/delta-spark-ut/known-failures.txt` | Committed baseline of currently-expected failures (`#` comments per line). |
| `.github/workflows/util/delta-spark-ut/README.md` | Documents the gate, bootstrapping, and baseline refresh. |
### Operational hardening
- **JDK 17 + Arrow/Netty**: forked test JVMs get the `--add-opens` set plus `-Dio.netty.tryReflectionSetAccessible=true` (otherwise Arrow's allocator fails to initialize).
- **Heap tuning**: forked-test heap and the sbt launcher's idle G1 behavior are tuned to keep the ~16 GB runner under the cgroup OOM threshold.
- **Hang watchdog**: a per-shard watchdog dumps threads and kills a forked test JVM that has gone silent too long, so a wedged suite can't stall the whole job.
- **DeletionVectorsSuite 2B-row tests**: two tests build/read/delete a 2-billion-row table and balloon the fork to ~13 GB of native memory (Velox row-index materialization), OOM-killing it and hanging the shard. They are force-failed (with a clear message) rather than silently ignored, so the gap stays visible until the native memory blow-up is fixed.
### Scope / known limitations
- Velox backend, x86 only; Delta `v4.2.0` / Spark 4.1 / Scala 2.13 / JDK 17.
- The baseline reflects the *current* set of known Delta-on-Gluten failures; refresh it via a `workflow_dispatch` run with `update_baseline=true`.
- **Future work -- Delta 4.3.0**: attempted, but the bundle (compiled against Delta 4.1.0) hits a binary-incompatible Delta change (`IdentityColumn.logTableWrite` first param `Snapshot` -> `SnapshotDescriptor`), which `NoSuchMethodError`s on every write. Supporting 4.3.0 needs the bundle built against 4.3.0; tracked as follow-up.
## How was this patch tested?
This change *is* CI. The Delta suite runs as part of `velox_backend_x86.yml` -- on every PR/trigger that touches Velox/core/cpp or the Delta CI files -- and via manual `workflow_dispatch`. In the latest runs all shards pass against the committed baseline (failures limited to known-failures entries; no regressions).
19,073 Delta tests run (18,297 passed / 776 failed).
### Main failures (776 baseline):
- 226 tests - Increment Metric: known issue https://github.com/apache/gluten/issues/9003. [Test with increment metric offload disabled](https://github.com/apache/gluten/actions/runs/28226442887/job/83623837492?pr=12380)
- 99 tests - VariantType - java.lang.UnsupportedOperationException: Unsupported data type: variant - Arrow throws (SparkArrowUtil.scala:60)
- ~47 tests - ClassCast ProjectExec -> WholeStageTransformer (Delta stats) - This will be addressed by https://github.com/apache/gluten/issues/11622#issuecomment-4421317668 `timestamp -> timestamp_ntz`
**Fixed since the first draft (#12371):** the 187 `MatchError List()` DataSkipping-empty-stats failures (caused by a `FileSourceScanExec` match) were fixed by cherry-picking the merged Delta PRs 7104 + 7105 (`FileSourceScanExec` -> `FileSourceScanLike`) during test setup. That dropped the baseline from 963 to **776** known failures (187 now-passing removed, 0 regressions).
## Delta Spark UT (Gluten) -- shard count vs test parallelism
Sharding is by **suite** (`MurmurHash3(suiteName) % NUM_SHARDS`), so total test work is fixed (~1250 fork-minutes). The runners are 4-core / ~16 GB. The committed config is **4 shards x 4 forks**.
| Config | Runner jobs | Forks/shard | Max shard | Wall-clock | Billed job-hrs* | Outcome |
|---|---|---|---|---|---|---|
| 16 shards x 1 fork | 16 | 1 | ~110 min | ~130 min | ~29 | green |
| **4 shards x 4 forks** | **4** | **4** | **158 min** | **178 min** | **~10.5** | **green** |
| 4 shards x 1 fork | 4 | 1 | 360 min (hit cap) | -- | -- | cancelled |
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3502475814
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -93,6 +102,24 @@ jobs:
name: velox-native-lib-centos-7-${{github.sha}}
path: ./cpp/build/
if-no-files-found: error
+ # Consumed by the reusable Delta workflow (delta-spark-ut job).
+ - uses: actions/upload-artifact@v4
+ with:
+ name: velox-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ # Delta Spark UT, run via the reusable workflow so it reuses the native lib +
+ # arrow jars built above instead of duplicating the native build. Not gated on
+ # Delta-only paths: core/velox/substrait/cpp/shims changes can affect Delta
+ # query offload, so this runs on every trigger like the other spark-test jobs.
+ delta-spark-ut:
+ needs: build-native-lib-centos-7
+ uses: ./.github/workflows/delta_spark_ut.yml
+ with:
+ native_lib_artifact: velox-native-lib-centos-7-${{ github.sha }}
+ arrow_jars_artifact: velox-arrow-jars-centos-7-${{ github.sha }}
+ secrets: inherit
Review Comment:
Done in 6fe545120. Removed secrets: inherit from the reusable Delta UT call. delta_spark_ut.yml declares no secrets: and references no ${{ secrets.* }} , and artifact download/cache use the automatic GITHUB_TOKEN (provided to reusable workflows without inherit ), so dropping it is safe and reduces blast radius.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3549649640
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,692 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. `spark_version` is the single source of truth for the
+ # Spark version: it drives the Gluten bundle profile (-Pspark-<v>), the bundle
+ # jar name, and Delta's -DsparkVersion, so the tests always run against a bundle
+ # built for the same Spark version (no separate value to keep in sync). Scala
+ # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from
+ # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a
+ # compatible delta_ref.
+ GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version }}
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version }}
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Every input has a default (workflow_call + workflow_dispatch), so they
+ # are always set; just surface them as step outputs for the steps below.
+ env:
+ DELTA_REF: ${{ inputs.delta_ref }}
+ SPARK_VERSION: ${{ inputs.spark_version }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline }}
+ FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
+
+ - name: Download Gluten bundle jar
+ uses: actions/download-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: gluten-bundle
+
+ - name: Install minimal tools
+ run: |
+ set -euo pipefail
+ # apache/gluten:centos-9-jdk17 already has java-17, git, tar, a POSIX
+ # shell, and curl-minimal (which provides the `curl` command sbt's
+ # launcher needs). Install the rest of what Delta's build/sbt and the
+ # tests may need. We deliberately do NOT install the full `curl`
+ # package -- it conflicts with the pre-installed curl-minimal.
+ yum install -y java-17-openjdk-devel which findutils gzip python3
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ git --version
+ curl --version | head -n 1
+
+ - name: Cache sbt / Ivy / Coursier
+ uses: actions/cache@v4
+ with:
+ path: |
+ /root/.sbt
+ /root/.ivy2
+ /root/.cache/coursier
+ # Intentionally NOT keyed by ${{ matrix.shard }} -- all shards share
+ # the same dependency tree, so a single shared cache (with parallel
+ # save races resolved by GH on a first-write-wins basis) gives the
+ # best storage / hit-rate tradeoff.
+ key: delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-${{ env.DELTA_SCALA_VERSION }}
+ restore-keys: |
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-
+
+ - name: Clone and patch Delta
+ run: |
+ set -euo pipefail
+ # `2>/dev/null ... || true` keeps a no-match `ls` from aborting the step
+ # under `set -o pipefail`, so the explicit check below emits a clear
+ # error instead of a generic "cannot access".
+ GLUTEN_JAR=$(ls "$GITHUB_WORKSPACE"/gluten-bundle/gluten-velox-bundle-spark*_*-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$GLUTEN_JAR" ] || [ ! -f "$GLUTEN_JAR" ]; then
+ echo "ERROR: No Gluten bundle jar found under $GITHUB_WORKSPACE/gluten-bundle/" >&2
+ ls -la "$GITHUB_WORKSPACE/gluten-bundle/" || true
+ exit 1
+ fi
+ echo "Using Gluten bundle: $GLUTEN_JAR"
+ bash "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/setup-delta.sh" \
+ "${{ steps.resolve.outputs.delta_ref }}" \
+ "$GITHUB_WORKSPACE/delta" \
+ "$GLUTEN_JAR" \
+ "$GITHUB_WORKSPACE"
+
+ - name: Run Delta spark module tests (shard ${{ matrix.shard }} / ${{ env.DELTA_NUM_SHARDS }})
+ env:
+ NUM_SHARDS: ${{ env.DELTA_NUM_SHARDS }}
+ TEST_PARALLELISM_COUNT: ${{ steps.resolve.outputs.test_parallelism }}
+ # Required by Delta to enable testing-only code paths
+ # (see delta build.sbt: "Test / envVars += DELTA_TESTING -> 1").
+ DELTA_TESTING: '1'
+ # JDK 17 + Gluten/Arrow/Netty requires extra --add-opens and the
+ # `io.netty.tryReflectionSetAccessible` system property; otherwise
+ # the forked test JVM fails with
+ # java.lang.UnsupportedOperationException: sun.misc.Unsafe or
+ # java.nio.DirectByteBuffer.<init>(long, int) not available
+ # as soon as Gluten's bundled Arrow allocator initializes Netty
+ # direct buffers. Delta's own `Test / javaOptions` (see
+ # project/CrossSparkVersions.scala `java17TestSettings`) sets the
+ # base add-opens but NOT the Netty property -- Delta's own tests
+ # don't load Arrow/Netty buffers in a way that triggers it.
+ #
+ # Use JAVA_TOOL_OPTIONS so the flags propagate to BOTH the sbt
+ # launcher JVM and the forked test JVM (sbt forks tests and the
+ # child inherits the parent's env). The set below mirrors
+ # `extraJavaTestArgs` from Gluten's own root pom.xml (the
+ # canonical Gluten test JVM flag set).
+ #
+ # NOTE: we deliberately do NOT put `-Xmx` here. JAVA_TOOL_OPTIONS
+ # is processed BEFORE the JVM command line, so Delta's explicit
+ # `-Xmx1024m` (set in build.sbt `Test / javaOptions`) would still
+ # win (last `-Xmx` wins). The forked-test-JVM heap is bumped via
+ # an sbt `set spark / Test / javaOptions ++= ...` command below,
+ # which APPENDS to Delta's own seq -- so our `-Xmx` lands AFTER
+ # `-Xmx1024m` and wins.
+ JAVA_TOOL_OPTIONS: >-
+ -XX:+IgnoreUnrecognizedVMOptions
+ --add-opens=java.base/java.lang=ALL-UNNAMED
+ --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
+ --add-opens=java.base/java.lang.reflect=ALL-UNNAMED
+ --add-opens=java.base/java.io=ALL-UNNAMED
+ --add-opens=java.base/java.net=ALL-UNNAMED
+ --add-opens=java.base/java.nio=ALL-UNNAMED
+ --add-opens=java.base/java.util=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
+ --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.cs=ALL-UNNAMED
+ --add-opens=java.base/sun.security.action=ALL-UNNAMED
+ --add-opens=java.base/sun.util.calendar=ALL-UNNAMED
+ -Djdk.reflect.useDirectMethodHandle=false
+ -Dio.netty.tryReflectionSetAccessible=true
+ -Dfile.encoding=UTF-8
+ run: |
+ set -euo pipefail
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ cd "$GITHUB_WORKSPACE/delta"
+ chmod +x build/sbt
+ # Only run the unified `spark` sbt project, NOT `sparkGroup/test` --
+ # `sparkGroup` aggregates many other projects (sparkV2, contribs,
+ # sharing, connect*, ...) that are out of scope for this pipeline.
+ #
+ # JVM heap layout -- two memory consumers on the ~16G runner:
+ # * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to
+ # return idle memory during the (long) test phase via G1 periodic GC
+ # (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the
+ # busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent
+ # forces each periodic GC to a full STW collection) that uncommits to a
+ # tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low
+ # -Xms512m floor.
+ # Without this the idle launcher holds ~5.3G for the whole run; with
+ # it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime
+ # config, so they cannot affect the measured pass/fail signal.
+ # * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command
+ # below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends
+ # so our -Xmx2G comes last and wins. Gluten offloads data to Velox
+ # off-heap (capped at 2g via spark.memory.offHeap.size in the patched
+ # DeltaSQLCommandTest), so the fork's heap need is modest. A larger
+ # fork heap pushed the cgroup peak past the ~16G OOM threshold and the
+ # kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G
+ # keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is
+ # analyzable.
+ # `-u target/test-reports` enables ScalaTest's JUnit XML reporter so
+ # every suite writes per-test results. Delta itself only configures
+ # the console reporter (-oDF), so without this we'd have no machine-
+ # readable results to gate on. The path is relative to the forked
+ # test JVM's working dir (Test / baseDirectory = spark/), i.e.
+ # delta/spark/target/test-reports/TEST-*.xml.
+ #
+ # We deliberately do NOT let an sbt non-zero exit (which fires on the
+ # MANY expected Delta-on-Gluten failures) fail this step directly.
+ # Instead the known-failures gate below decides pass/fail: the build
+ # is green when the only failures are ones already recorded in the
+ # baseline, and red on a genuine regression.
+ set +e
+ # --- hang watchdog ---------------------------------------------------
+ # Shard 2 (and occasionally others) hangs indefinitely after a suite's
+ # last test with no further output. ScalaTest's failAfter only wraps
+ # individual test BODIES, so a wedge in suite teardown/afterAll -- or in
+ # a non-interruptible native Velox/JNI call that ignores
+ # Thread.interrupt() -- has no timeout and stalls until the 350-min job
+ # limit with zero diagnostics. This watchdog dumps the forked test JVM's
+ # threads (to the job log, and to a file for the artifact) once the test
+ # output has been silent for too long, so the deadlock is diagnosable.
+ SBT_LOG="/tmp/sbt-spark-test-shard-${{ matrix.shard }}.log"
+ : > "$SBT_LOG"
+ rm -f /tmp/sbt-done
+ (
+ # CRITICAL: the step shell runs with `bash -eo pipefail`, which the
+ # subshell inherits. Without `set +e` here, ANY non-zero command --
+ # e.g. fork detection finding no match, or `kill`/`jps` returning
+ # non-zero -- silently kills this watchdog. That errexit kill (plus a
+ # /proc detection miss) once made the watchdog capture ZERO dumps. A
+ # diagnostic must never abort on a failed probe.
+ set +e +o pipefail
+ JSTACK="${JAVA_HOME}/bin/jstack"
+ JPS="${JAVA_HOME}/bin/jps"
+ silent_limit=900 # 15 min with no new test output => treat as hung
+ dumps=0
+ fork_pids() {
+ # The sbt test fork's main class is sbt.ForkMain. Prefer jps (reads
+ # the main class from hsperfdata, robust to sbt's @argfile launch);
+ # fall back to scanning /proc cmdline + @argfile.
+ "$JPS" -l 2>/dev/null | awk '/sbt\.ForkMain/ {print $1}'
+ local p cl arg
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] || continue
+ cl="$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)"
+ case "$cl" in *sbt.ForkMain*) echo "${p##*/}"; continue ;; esac
+ arg="$(printf '%s' "$cl" | tr ' ' '\n' | sed -n 's/^@//p' | head -1)"
+ [ -n "$arg" ] && [ -f "$arg" ] && grep -qa 'sbt\.ForkMain' "$arg" 2>/dev/null \
+ && echo "${p##*/}"
+ done
+ }
+ all_java_pids() {
+ "$JPS" -q 2>/dev/null
+ local p
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] && echo "${p##*/}"
+ done
+ }
+ echo "HANG WATCHDOG armed: dumps the test JVM after ${silent_limit}s of output silence"
+ hb=0
+ while [ ! -f /tmp/sbt-done ]; do
+ sleep 60
+ [ -f "$SBT_LOG" ] || continue
+ now=$(date +%s)
+ mtime=$(stat -c %Y "$SBT_LOG" 2>/dev/null || echo "$now")
+ silent=$(( now - mtime ))
+ # Per-minute memory profile: heap tuning proved the ~16G OOM peak is
+ # NATIVE-driven, so log which JVM (sbt launcher vs fork) actually grows
+ # toward it -- the last lines before a hang reveal the real hog to cut.
+ # Read /proc directly (no `ps` dependency in the minimal container).
+ memnow=$(awk '{printf "%.2fG",$1/1073741824}' /sys/fs/cgroup/memory.current 2>/dev/null)
+ jvmrss=""
+ for mp in $(all_java_pids 2>/dev/null | sort -un); do
+ r=$(awk '/^VmRSS:/{print $2}' "/proc/$mp/status" 2>/dev/null)
+ [ -n "$r" ] && jvmrss="$jvmrss $(( r / 1024 ))M(p$mp)"
+ done
+ echo "MEM cgroup=${memnow} JVMs=[${jvmrss# }]"
+ hb=$(( hb + 1 ))
+ # Heartbeat every ~5 min so we can SEE the watchdog is alive (and how
+ # long the test has been silent) without waiting for a hang.
+ [ $(( hb % 5 )) -eq 0 ] && echo "HANG WATCHDOG: alive; last test output ${silent}s ago"
+ if [ "$silent" -ge "$silent_limit" ] && [ "$dumps" -lt 3 ]; then
+ dumps=$(( dumps + 1 ))
+ pids="$(fork_pids | sort -un)"
+ # Safety net: if the fork JVM cannot be pinpointed, dump EVERY JVM.
+ [ -n "$pids" ] || pids="$(all_java_pids | sort -un)"
+ echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $pids))"
+ [ -n "$pids" ] || echo "HANG WATCHDOG: no java process found to dump"
+ for pid in $pids; do
+ # SIGQUIT makes the JVM print a full thread dump to its OWN stderr,
+ # which sbt relays into the test log via the SAME stream as test
+ # output -- so it lands in the job log even when a separately
+ # spawned jstack child's output would be buffered/lost. Also write
+ # jstack to a file for the per-shard artifact.
+ echo "----- SIGQUIT + jstack pid ${pid} -----"
+ kill -QUIT "$pid" 2>/dev/null || echo "HANG WATCHDOG: kill -QUIT failed for pid ${pid}"
+ timeout 120 "$JSTACK" -l "$pid" > "/tmp/threaddump-shard-${{ matrix.shard }}-${dumps}-${pid}.txt" 2>&1 \
+ || echo "HANG WATCHDOG: jstack failed/timed out for pid ${pid}"
+ done
+ echo "::endgroup::"
+ # The dump is now captured (job log via SIGQUIT + artifact via
+ # jstack file). A hung JVM otherwise stalls the whole shard until
+ # the 350-min job timeout AND keeps the job log frozen so the dump
+ # never becomes reachable. So KILL the wedged JVM(s): the suite
+ # fails fast (acceptable -- errors are expected; only an
+ # unrecoverable hang blocks CI), the job proceeds/ends, and the log
+ # + artifacts flush. Give SIGQUIT a moment to print first.
+ sleep 20
+ echo "HANG WATCHDOG: killing wedged JVM(s) to unblock the shard: $(printf '%s ' $pids)"
+ for pid in $pids; do kill -KILL "$pid" 2>/dev/null; done
+ fi
+ done
+ ) &
+ WATCHDOG_PID=$!
+
+ ./build/sbt \
+ -DsparkVersion=${{ steps.resolve.outputs.spark_version }} \
+ -v \
+ -J-XX:+UseG1GC -J-Xms512m -J-Xmx4G \
+ -J-XX:G1PeriodicGCInterval=10000 \
+ -J-XX:G1PeriodicGCSystemLoadThreshold=0 \
+ -J-XX:-G1PeriodicGCInvokesConcurrent \
+ -J-XX:MinHeapFreeRatio=5 -J-XX:MaxHeapFreeRatio=15 \
+ "++ ${DELTA_SCALA_VERSION}" \
+ 'set spark / Test / javaOptions ++= Seq("-Xmx2G", "-XX:+HeapDumpOnOutOfMemoryError", "-XX:HeapDumpPath=/tmp/")' \
+ 'set spark / Test / testOptions += Tests.Argument(TestFrameworks.ScalaTest, "-u", "target/test-reports")' \
+ "spark/test" 2>&1 | tee "$SBT_LOG"
+ SBT_EXIT=${PIPESTATUS[0]}
+ touch /tmp/sbt-done
+ kill "$WATCHDOG_PID" 2>/dev/null || true
+ set -e
+ echo "sbt spark/test exited with ${SBT_EXIT}"
+
+ # Memory forensics: a sudden forked-JVM death with no hs_err and no heap
+ # dump is almost always a kernel/cgroup OOM-kill (Velox off-heap + JVM
+ # heap exceeding the ~16G runner). Surface the cgroup peak + oom_kill
+ # count so we can confirm/measure it (cgroup v2 paths; best-effort).
+ ( echo "=== cgroup memory forensics (exit ${SBT_EXIT}) ==="
+ for f in /sys/fs/cgroup/memory.peak /sys/fs/cgroup/memory.max \
+ /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.events; do
+ [ -r "$f" ] && { echo "--- $f ---"; cat "$f"; }
+ done ) || true
+
+ # A compile/launch failure leaves no reports at all. In that case the
+ # gate would see zero failures and pass spuriously, so fail loudly.
+ REPORT_COUNT=$(find . -path '*/target/test-reports/*.xml' 2>/dev/null | wc -l || true)
+ echo "Found ${REPORT_COUNT} JUnit XML report file(s)."
+ if [ "${REPORT_COUNT}" -eq 0 ]; then
+ echo "::error::sbt produced no test reports (exit ${SBT_EXIT}) -- likely a compile or launch failure, not test failures."
+ exit 1
+ fi
+
+ # update_baseline=true -> SEED mode (record failures, never fail) so the
+ # baseline can be (re)generated. Otherwise ENFORCE against the baseline.
+ GATE_MODE=enforce
+ if [ "${{ steps.resolve.outputs.update_baseline }}" = "true" ]; then
+ GATE_MODE=seed
+ fi
+ mkdir -p "$GITHUB_WORKSPACE/gate-out"
+ python3 "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/compare-test-results.py" \
+ --mode "${GATE_MODE}" \
+ --reports-dir "$GITHUB_WORKSPACE/delta" \
+ --known-failures "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/known-failures.txt" \
+ --flaky-tests "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/flaky-tests.txt" \
+ --failures-out "$GITHUB_WORKSPACE/gate-out/failures-shard-${{ matrix.shard }}.txt" \
+ --ran-out "$GITHUB_WORKSPACE/gate-out/ran-shard-${{ matrix.shard }}.txt" \
+ --fail-on-fixed "${{ steps.resolve.outputs.fail_on_fixed }}"
Review Comment:
Fixed in 1409aec23cab3f317e5b9723fe8381bfe5d4cad8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-4973074034
Thanks.
It is not a blocker for this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5243269976
@zhztheplayer, @philo-he does it also need @zhouyuan approval before merging it?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3610335542
##########
.github/workflows/util/delta-spark-ut/run-delta-tests.sh:
##########
@@ -0,0 +1,240 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Runs the Delta `spark` module tests for one shard under the Gluten bundle:
+# arms a hang watchdog (thread-dumps + kills a wedged fork), invokes sbt
+# spark/test with the tuned JVM/heap flags, prints cgroup memory forensics, and
+# then gates the results against the baseline (compare-test-results.py). Extracted
+# from delta_spark_ut.yml so the workflow step stays readable.
+#
+# Driven by environment (set by the workflow step / job):
+# SHARD_ID - this shard's id (matrix.shard)
+# SPARK_VERSION - Delta -DsparkVersion value
+# UPDATE_BASELINE - 'true' -> gate seed mode; else enforce
+# FAIL_ON_FIXED - passed through to the gate
+# DELTA_SCALA_VERSION, NUM_SHARDS, TEST_PARALLELISM_COUNT, DELTA_TESTING
+# - test env (see the workflow step's `env:` block)
+# GITHUB_WORKSPACE - repo root (holds the Delta clone + util scripts)
+#
+# JAVA_TOOL_OPTIONS is set by sourcing java-test-args.sh (below), not the caller.
+
+set -euo pipefail
+export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+export PATH=$JAVA_HOME/bin:$PATH
+# Gluten/JDK17 test JVM flags (--add-opens + Netty property), shared with local
+# dev runs. Sets JAVA_TOOL_OPTIONS so it reaches the sbt launcher + forked JVMs.
+# shellcheck source=./java-test-args.sh
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/java-test-args.sh"
+cd "$GITHUB_WORKSPACE/delta"
+chmod +x build/sbt
+# Only run the unified `spark` sbt project, NOT `sparkGroup/test` --
+# `sparkGroup` aggregates many other projects (sparkV2, contribs,
+# sharing, connect*, ...) that are out of scope for this pipeline.
+#
+# JVM heap layout -- two memory consumers on the ~16G runner:
+# * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to
+# return idle memory during the (long) test phase via G1 periodic GC
+# (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the
+# busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent
+# forces each periodic GC to a full STW collection) that uncommits to a
+# tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low
+# -Xms512m floor.
+# Without this the idle launcher holds ~5.3G for the whole run; with
+# it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime
+# config, so they cannot affect the measured pass/fail signal.
+# * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command
+# below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends
+# so our -Xmx2G comes last and wins. Gluten offloads data to Velox
+# off-heap (capped at 2g via spark.memory.offHeap.size in the patched
+# DeltaSQLCommandTest), so the fork's heap need is modest. A larger
+# fork heap pushed the cgroup peak past the ~16G OOM threshold and the
+# kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G
+# keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is
+# analyzable.
+# `-u target/test-reports` enables ScalaTest's JUnit XML reporter so
+# every suite writes per-test results. Delta itself only configures
+# the console reporter (-oDF), so without this we'd have no machine-
+# readable results to gate on. The path is relative to the forked
+# test JVM's working dir (Test / baseDirectory = spark/), i.e.
+# delta/spark/target/test-reports/TEST-*.xml.
+#
+# We deliberately do NOT let an sbt non-zero exit (which fires on the
+# MANY expected Delta-on-Gluten failures) fail this step directly.
+# Instead the known-failures gate below decides pass/fail: the build
+# is green when the only failures are ones already recorded in the
+# baseline, and red on a genuine regression.
+set +e
+# --- hang watchdog ---------------------------------------------------
+# Shard 2 (and occasionally others) hangs indefinitely after a suite's
+# last test with no further output. ScalaTest's failAfter only wraps
+# individual test BODIES, so a wedge in suite teardown/afterAll -- or in
+# a non-interruptible native Velox/JNI call that ignores
+# Thread.interrupt() -- has no timeout and stalls until the 350-min job
+# limit with zero diagnostics. This watchdog dumps the forked test JVM's
+# threads (to the job log, and to a file for the artifact) once the test
+# output has been silent for too long, so the deadlock is diagnosable.
+SBT_LOG="/tmp/sbt-spark-test-shard-${SHARD_ID}.log"
+: > "$SBT_LOG"
+rm -f /tmp/sbt-done
+(
+ # CRITICAL: the step shell runs with `bash -eo pipefail`, which the
+ # subshell inherits. Without `set +e` here, ANY non-zero command --
+ # e.g. fork detection finding no match, or `kill`/`jps` returning
+ # non-zero -- silently kills this watchdog. That errexit kill (plus a
+ # /proc detection miss) once made the watchdog capture ZERO dumps. A
+ # diagnostic must never abort on a failed probe.
+ set +e +o pipefail
+ JSTACK="${JAVA_HOME}/bin/jstack"
+ JPS="${JAVA_HOME}/bin/jps"
+ silent_limit=900 # 15 min with no new test output => treat as hung
+ dumps=0
+ fork_pids() {
+ # The sbt test fork's main class is sbt.ForkMain. Prefer jps (reads
+ # the main class from hsperfdata, robust to sbt's @argfile launch);
+ # fall back to scanning /proc cmdline + @argfile.
+ "$JPS" -l 2>/dev/null | awk '/sbt\.ForkMain/ {print $1}'
+ local p cl arg
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] || continue
+ cl="$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)"
+ case "$cl" in *sbt.ForkMain*) echo "${p##*/}"; continue ;; esac
+ arg="$(printf '%s' "$cl" | tr ' ' '\n' | sed -n 's/^@//p' | head -1)"
+ [ -n "$arg" ] && [ -f "$arg" ] && grep -qa 'sbt\.ForkMain' "$arg" 2>/dev/null \
+ && echo "${p##*/}"
+ done
+ }
+ all_java_pids() {
+ "$JPS" -q 2>/dev/null
+ local p
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] && echo "${p##*/}"
+ done
+ }
+ echo "HANG WATCHDOG armed: dumps the test JVM after ${silent_limit}s of output silence"
+ hb=0
+ while [ ! -f /tmp/sbt-done ]; do
+ sleep 60
+ [ -f "$SBT_LOG" ] || continue
+ now=$(date +%s)
+ mtime=$(stat -c %Y "$SBT_LOG" 2>/dev/null || echo "$now")
+ silent=$(( now - mtime ))
+ # Per-minute memory profile: heap tuning proved the ~16G OOM peak is
+ # NATIVE-driven, so log which JVM (sbt launcher vs fork) actually grows
+ # toward it -- the last lines before a hang reveal the real hog to cut.
+ # Read /proc directly (no `ps` dependency in the minimal container).
+ memnow=$(awk '{printf "%.2fG",$1/1073741824}' /sys/fs/cgroup/memory.current 2>/dev/null)
+ jvmrss=""
+ for mp in $(all_java_pids 2>/dev/null | sort -un); do
+ r=$(awk '/^VmRSS:/{print $2}' "/proc/$mp/status" 2>/dev/null)
+ [ -n "$r" ] && jvmrss="$jvmrss $(( r / 1024 ))M(p$mp)"
+ done
+ echo "MEM cgroup=${memnow} JVMs=[${jvmrss# }]"
+ hb=$(( hb + 1 ))
+ # Heartbeat every ~5 min so we can SEE the watchdog is alive (and how
+ # long the test has been silent) without waiting for a hang.
+ [ $(( hb % 5 )) -eq 0 ] && echo "HANG WATCHDOG: alive; last test output ${silent}s ago"
+ if [ "$silent" -ge "$silent_limit" ] && [ "$dumps" -lt 3 ]; then
+ dumps=$(( dumps + 1 ))
+ pids="$(fork_pids | sort -un)"
+ # Safety net: if the fork JVM cannot be pinpointed, dump EVERY JVM.
+ [ -n "$pids" ] || pids="$(all_java_pids | sort -un)"
+ echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $pids))"
+ [ -n "$pids" ] || echo "HANG WATCHDOG: no java process found to dump"
+ for pid in $pids; do
+ # SIGQUIT makes the JVM print a full thread dump to its OWN stderr,
+ # which sbt relays into the test log via the SAME stream as test
+ # output -- so it lands in the job log even when a separately
+ # spawned jstack child's output would be buffered/lost. Also write
+ # jstack to a file for the per-shard artifact.
+ echo "----- SIGQUIT + jstack pid ${pid} -----"
+ kill -QUIT "$pid" 2>/dev/null || echo "HANG WATCHDOG: kill -QUIT failed for pid ${pid}"
+ timeout 120 "$JSTACK" -l "$pid" > "/tmp/threaddump-shard-${SHARD_ID}-${dumps}-${pid}.txt" 2>&1 \
+ || echo "HANG WATCHDOG: jstack failed/timed out for pid ${pid}"
+ done
+ echo "::endgroup::"
+ # The dump is now captured (job log via SIGQUIT + artifact via
+ # jstack file). A hung JVM otherwise stalls the whole shard until
+ # the 350-min job timeout AND keeps the job log frozen so the dump
+ # never becomes reachable. So KILL the wedged JVM(s): the suite
+ # fails fast (acceptable -- errors are expected; only an
+ # unrecoverable hang blocks CI), the job proceeds/ends, and the log
+ # + artifacts flush. Give SIGQUIT a moment to print first.
+ sleep 20
+ echo "HANG WATCHDOG: killing wedged JVM(s) to unblock the shard: $(printf '%s ' $pids)"
+ for pid in $pids; do kill -KILL "$pid" 2>/dev/null; done
Review Comment:
One gap I think is worth closing here: when the watchdog kills a fork, the suite that was running plus everything queued behind it in that JVM never write any XML (the reporter runs inside the fork), and since we intentionally don't fail on sbt's exit code, the gate only judges the tests that produced reports. The skipped suites just vanish — they're not a regression, not now-passing, nothing. `REPORT_COUNT` won't catch it either because the earlier suites did write reports. So am I reading it right that a hang + kill would currently let the shard go green with those suites unrun? Your own comment in setup-delta.sh actually describes this ("every suite QUEUED AFTER it in that fork is skipped") — it's just not reflected in the gate result yet.
If so, a cheap fix that keeps the kill-to-unblock behavior: touch a marker file here when the watchdog kills, and after sbt returns, fail the shard if the marker exists. A watchdog kill is already an abnormal run, it deserves a red X.
##########
.github/workflows/util/delta-spark-ut/run-delta-tests.sh:
##########
@@ -0,0 +1,240 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Runs the Delta `spark` module tests for one shard under the Gluten bundle:
+# arms a hang watchdog (thread-dumps + kills a wedged fork), invokes sbt
+# spark/test with the tuned JVM/heap flags, prints cgroup memory forensics, and
+# then gates the results against the baseline (compare-test-results.py). Extracted
+# from delta_spark_ut.yml so the workflow step stays readable.
+#
+# Driven by environment (set by the workflow step / job):
+# SHARD_ID - this shard's id (matrix.shard)
+# SPARK_VERSION - Delta -DsparkVersion value
+# UPDATE_BASELINE - 'true' -> gate seed mode; else enforce
+# FAIL_ON_FIXED - passed through to the gate
+# DELTA_SCALA_VERSION, NUM_SHARDS, TEST_PARALLELISM_COUNT, DELTA_TESTING
+# - test env (see the workflow step's `env:` block)
+# GITHUB_WORKSPACE - repo root (holds the Delta clone + util scripts)
+#
+# JAVA_TOOL_OPTIONS is set by sourcing java-test-args.sh (below), not the caller.
+
+set -euo pipefail
+export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+export PATH=$JAVA_HOME/bin:$PATH
+# Gluten/JDK17 test JVM flags (--add-opens + Netty property), shared with local
+# dev runs. Sets JAVA_TOOL_OPTIONS so it reaches the sbt launcher + forked JVMs.
+# shellcheck source=./java-test-args.sh
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/java-test-args.sh"
+cd "$GITHUB_WORKSPACE/delta"
+chmod +x build/sbt
+# Only run the unified `spark` sbt project, NOT `sparkGroup/test` --
+# `sparkGroup` aggregates many other projects (sparkV2, contribs,
+# sharing, connect*, ...) that are out of scope for this pipeline.
+#
+# JVM heap layout -- two memory consumers on the ~16G runner:
+# * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to
+# return idle memory during the (long) test phase via G1 periodic GC
+# (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the
+# busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent
+# forces each periodic GC to a full STW collection) that uncommits to a
+# tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low
+# -Xms512m floor.
+# Without this the idle launcher holds ~5.3G for the whole run; with
+# it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime
+# config, so they cannot affect the measured pass/fail signal.
+# * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command
+# below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends
+# so our -Xmx2G comes last and wins. Gluten offloads data to Velox
+# off-heap (capped at 2g via spark.memory.offHeap.size in the patched
+# DeltaSQLCommandTest), so the fork's heap need is modest. A larger
+# fork heap pushed the cgroup peak past the ~16G OOM threshold and the
+# kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G
+# keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is
+# analyzable.
+# `-u target/test-reports` enables ScalaTest's JUnit XML reporter so
+# every suite writes per-test results. Delta itself only configures
+# the console reporter (-oDF), so without this we'd have no machine-
+# readable results to gate on. The path is relative to the forked
+# test JVM's working dir (Test / baseDirectory = spark/), i.e.
+# delta/spark/target/test-reports/TEST-*.xml.
+#
+# We deliberately do NOT let an sbt non-zero exit (which fires on the
+# MANY expected Delta-on-Gluten failures) fail this step directly.
+# Instead the known-failures gate below decides pass/fail: the build
+# is green when the only failures are ones already recorded in the
+# baseline, and red on a genuine regression.
+set +e
+# --- hang watchdog ---------------------------------------------------
+# Shard 2 (and occasionally others) hangs indefinitely after a suite's
+# last test with no further output. ScalaTest's failAfter only wraps
+# individual test BODIES, so a wedge in suite teardown/afterAll -- or in
+# a non-interruptible native Velox/JNI call that ignores
+# Thread.interrupt() -- has no timeout and stalls until the 350-min job
+# limit with zero diagnostics. This watchdog dumps the forked test JVM's
+# threads (to the job log, and to a file for the artifact) once the test
+# output has been silent for too long, so the deadlock is diagnosable.
+SBT_LOG="/tmp/sbt-spark-test-shard-${SHARD_ID}.log"
+: > "$SBT_LOG"
+rm -f /tmp/sbt-done
+(
+ # CRITICAL: the step shell runs with `bash -eo pipefail`, which the
+ # subshell inherits. Without `set +e` here, ANY non-zero command --
+ # e.g. fork detection finding no match, or `kill`/`jps` returning
+ # non-zero -- silently kills this watchdog. That errexit kill (plus a
+ # /proc detection miss) once made the watchdog capture ZERO dumps. A
+ # diagnostic must never abort on a failed probe.
+ set +e +o pipefail
+ JSTACK="${JAVA_HOME}/bin/jstack"
+ JPS="${JAVA_HOME}/bin/jps"
+ silent_limit=900 # 15 min with no new test output => treat as hung
+ dumps=0
+ fork_pids() {
+ # The sbt test fork's main class is sbt.ForkMain. Prefer jps (reads
+ # the main class from hsperfdata, robust to sbt's @argfile launch);
+ # fall back to scanning /proc cmdline + @argfile.
+ "$JPS" -l 2>/dev/null | awk '/sbt\.ForkMain/ {print $1}'
+ local p cl arg
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] || continue
+ cl="$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)"
+ case "$cl" in *sbt.ForkMain*) echo "${p##*/}"; continue ;; esac
+ arg="$(printf '%s' "$cl" | tr ' ' '\n' | sed -n 's/^@//p' | head -1)"
+ [ -n "$arg" ] && [ -f "$arg" ] && grep -qa 'sbt\.ForkMain' "$arg" 2>/dev/null \
+ && echo "${p##*/}"
+ done
+ }
+ all_java_pids() {
+ "$JPS" -q 2>/dev/null
+ local p
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] && echo "${p##*/}"
+ done
+ }
+ echo "HANG WATCHDOG armed: dumps the test JVM after ${silent_limit}s of output silence"
+ hb=0
+ while [ ! -f /tmp/sbt-done ]; do
+ sleep 60
+ [ -f "$SBT_LOG" ] || continue
+ now=$(date +%s)
+ mtime=$(stat -c %Y "$SBT_LOG" 2>/dev/null || echo "$now")
+ silent=$(( now - mtime ))
+ # Per-minute memory profile: heap tuning proved the ~16G OOM peak is
+ # NATIVE-driven, so log which JVM (sbt launcher vs fork) actually grows
+ # toward it -- the last lines before a hang reveal the real hog to cut.
+ # Read /proc directly (no `ps` dependency in the minimal container).
+ memnow=$(awk '{printf "%.2fG",$1/1073741824}' /sys/fs/cgroup/memory.current 2>/dev/null)
+ jvmrss=""
+ for mp in $(all_java_pids 2>/dev/null | sort -un); do
+ r=$(awk '/^VmRSS:/{print $2}' "/proc/$mp/status" 2>/dev/null)
+ [ -n "$r" ] && jvmrss="$jvmrss $(( r / 1024 ))M(p$mp)"
+ done
+ echo "MEM cgroup=${memnow} JVMs=[${jvmrss# }]"
+ hb=$(( hb + 1 ))
+ # Heartbeat every ~5 min so we can SEE the watchdog is alive (and how
+ # long the test has been silent) without waiting for a hang.
+ [ $(( hb % 5 )) -eq 0 ] && echo "HANG WATCHDOG: alive; last test output ${silent}s ago"
+ if [ "$silent" -ge "$silent_limit" ] && [ "$dumps" -lt 3 ]; then
+ dumps=$(( dumps + 1 ))
+ pids="$(fork_pids | sort -un)"
+ # Safety net: if the fork JVM cannot be pinpointed, dump EVERY JVM.
+ [ -n "$pids" ] || pids="$(all_java_pids | sort -un)"
+ echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $pids))"
+ [ -n "$pids" ] || echo "HANG WATCHDOG: no java process found to dump"
Review Comment:
nit :Small risk with this fallback: before any `sbt.ForkMain` exists (resolving dependencies, or compiling the big spark test module on a cold cache), sbt can go quiet for a long time. If that quiet stretch hits 900s, the watchdog trips with no fork pids and this line makes it kill the sbt launcher itself — the job dies with a confusing "compile or launch failure" and the whole slot is wasted. Dumping all JVMs for diagnostics makes sense, but maybe only kill when we actually matched fork pids?
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,747 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline file exists but is empty (not yet bootstrapped) the mode
+ automatically degrades to ``seed`` so the first run is never spuriously red.
+ A *missing* ``--known-failures`` file is treated as a configuration error
+ (the gate fails) so a mis-referenced path can't silently pass.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard). Pass ``--expected-shards N``
+ to fail when fewer than ``N`` shards contributed gate lists (a shard that
+ died before writing them), so an incomplete baseline is never produced.
+
+Flaky quarantine (``--flaky-tests``)
+ Some Delta-on-Gluten failures are non-deterministic (e.g. a native bug that
+ only triggers on certain runtime plans), so they are neither a stable pass
+ nor a stable failure and cannot live in the baseline: baselining them turns
+ the gate red on every run where they pass, and leaving them out turns it red
+ on every run where they fail. ``flaky-tests.txt`` quarantines them -- a
+ quarantined test never counts as a regression (when it fails) nor as
+ now-passing (when it passes), and is excluded from the regenerated baseline.
+ Its SUITE is an fnmatch glob (so one line covers a root-cause family across
+ generated suite variants); its TEST name is matched exactly.
+
+Flaky quarantine by error signature (``--flaky-error-patterns``)
+ When a failure is caused by a known nondeterministic bug that surfaces on a
+ *different test each run* (e.g. the native Delta DV bitmap row-index error),
+ matching by test name is whack-a-mole. ``flaky-error-patterns.txt`` instead
+ quarantines by root cause: each line is a regex matched against a failed
+ test's <failure>/<error> text, and any failure that matches is treated as
+ flaky regardless of which test it landed on.
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import fnmatch
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+
+class NoReportsError(RuntimeError):
+ """Raised when no JUnit <testsuite> elements are found under reports_dir."""
+
+
+class CorruptReportError(NoReportsError):
+ """Raised when an expected JUnit report file (TEST-*.xml) fails to parse.
+
+ Subclasses NoReportsError so the enforce/seed handler treats a truncated
+ report as a hard data error (exit 2) instead of silently dropping the
+ suite's results and letting the gate pass on partial data.
+ """
+
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def make_is_flaky(flaky_entries):
+ """Build a predicate that matches a (suite, test) tuple against flaky entries.
+
+ A flaky entry quarantines a test whose failure is known to be non-deterministic
+ (see flaky-tests.txt). The entry's SUITE is treated as an fnmatch glob so a
+ single line can cover a root-cause family across generated suite variants
+ (e.g. ``*DVs*Suite`` matches every deletion-vector merge suite, ``*`` matches
+ any suite); the TEST name is matched exactly (test names are freeform and may
+ contain glob metacharacters, so they are never globbed).
+ """
+ exact = set()
+ globbed = []
+ for suite, test in flaky_entries:
+ if any(ch in suite for ch in "*?["):
+ globbed.append((suite, test))
+ else:
+ exact.add((suite, test))
+
+ def is_flaky(entry):
+ if entry in exact:
+ return True
+ suite, test = entry
+ for glob_suite, glob_test in globbed:
+ if test == glob_test and fnmatch.fnmatchcase(suite, glob_suite):
+ return True
+ return False
+
+ return is_flaky
+
+
+def load_patterns(path):
+ """Load flaky-error regex patterns from a file (one per line).
+
+ Blank lines and ``#`` comments are ignored. Each remaining line is compiled
+ as a case-sensitive regex. These match the FAILURE TEXT of a failed test
+ (its JUnit <failure>/<error> message + stack), so a test that fails with a
+ known-nondeterministic native error (e.g. the Delta DV bitmap row-index bug)
+ can be quarantined by root cause instead of by exact test name.
+ """
+ patterns = []
+ if not path or not os.path.exists(path):
+ return patterns
+ with open(path, encoding="utf-8") as fh:
+ for line in fh:
+ line = line.rstrip("\n")
+ if not line.strip() or line.lstrip().startswith("#"):
+ continue
+ patterns.append(re.compile(line))
+ return patterns
+
+
+def make_signature_matcher(patterns):
+ """Return a predicate matching a failure text against any flaky-error pattern."""
+
+ def matches(text):
+ if not text:
+ return False
+ return any(p.search(text) for p in patterns)
+
+ return matches
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def _failure_text(tc):
+ """Concatenate the message attribute + body text of a testcase's
+ <failure>/<error> children, for error-signature matching."""
+ parts = []
+ for c in tc:
+ if c.tag.split("}")[-1] in ("failure", "error"):
+ msg = c.get("message")
+ if msg:
+ parts.append(msg)
+ if c.text:
+ parts.append(c.text)
+ return "\n".join(parts)
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped, fail_texts). The first three are sets of
+ (suite, test) tuples; fail_texts maps each failed (suite, test) to its
+ combined <failure>/<error> message + stack text (used for error-signature
+ quarantine). A test is 'failed' if its <testcase> has a <failure> or <error>
+ child, 'skipped' if it has a <skipped> child, otherwise 'passed'. Suite-level
+ aborts (a <testsuite> reporting errors/failures with no failing <testcase>)
+ are recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+ fail_texts = {}
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ # A TEST-*.xml that fails to parse is almost always a report truncated
+ # when a forked test JVM was killed mid-write (e.g. OOM). Silently
+ # skipping it drops that suite's results and could let the gate go
+ # green on partial data, so fail hard for report files. Other XML that
+ # merely matched the broad `target/**` glob is still skipped.
+ if os.path.basename(xml_file).startswith("TEST-"):
+ raise CorruptReportError(
+ "corrupt or truncated JUnit report {}: {}. Refusing to "
+ "evaluate the gate on partial data.".format(xml_file, exc)
+ )
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = (suite, name)
Review Comment:
Nit:
Possible trap as these keys come straight from the XML, but the txt files get normalized on the way through (newlines collapsed on write, lines stripped on load). So if a test name ever has trailing whitespace or a newline in it, it'll fail, the gate will print a nice copy-pasteable `REGRESSION` line — and pasting that line into known-failures.txt will never suppress it, because loading strips it again and the keys don't match.
Nothing in the current baseline hits this, but Delta's test names are freeform enough that a version bump could. Fix is one line: run the XML-side name through the same normalization here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3487670744
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,672 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Delta `-DsparkVersion` value (must match the Gluten -P profile below)'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. Spark 4.1 + Scala 2.13 + JDK 17 matches Delta v4.2.0's
+ # default Spark version (4.1.0) from project/CrossSparkVersions.scala.
+ GLUTEN_SPARK_PROFILE: 'spark-4.1'
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: '4.1'
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix.
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar | head -n 1)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Every input has a default (workflow_call + workflow_dispatch), so they
+ # are always set; just surface them as step outputs for the steps below.
+ env:
+ DELTA_REF: ${{ inputs.delta_ref }}
+ SPARK_VERSION: ${{ inputs.spark_version }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline }}
+ FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
+
+ - name: Download Gluten bundle jar
+ uses: actions/download-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: gluten-bundle
+
+ - name: Install minimal tools
+ run: |
+ set -euo pipefail
+ # apache/gluten:centos-9-jdk17 already has java-17, git, tar, a POSIX
+ # shell, and curl-minimal (which provides the `curl` command sbt's
+ # launcher needs). Install the rest of what Delta's build/sbt and the
+ # tests may need. We deliberately do NOT install the full `curl`
+ # package -- it conflicts with the pre-installed curl-minimal.
+ yum install -y java-17-openjdk-devel which findutils gzip python3
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ git --version
+ curl --version | head -n 1
+
+ - name: Cache sbt / Ivy / Coursier
+ uses: actions/cache@v4
+ with:
+ path: |
+ /root/.sbt
+ /root/.ivy2
+ /root/.cache/coursier
+ # Intentionally NOT keyed by ${{ matrix.shard }} -- all shards share
+ # the same dependency tree, so a single shared cache (with parallel
+ # save races resolved by GH on a first-write-wins basis) gives the
+ # best storage / hit-rate tradeoff.
+ key: delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-${{ env.DELTA_SCALA_VERSION }}
+ restore-keys: |
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-
+
+ - name: Clone and patch Delta
+ run: |
+ set -euo pipefail
+ GLUTEN_JAR=$(ls "$GITHUB_WORKSPACE"/gluten-bundle/gluten-velox-bundle-spark*_*-linux_amd64-*.jar | head -n 1)
+ echo "Using Gluten bundle: $GLUTEN_JAR"
Review Comment:
Done
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,672 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Delta `-DsparkVersion` value (must match the Gluten -P profile below)'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. Spark 4.1 + Scala 2.13 + JDK 17 matches Delta v4.2.0's
+ # default Spark version (4.1.0) from project/CrossSparkVersions.scala.
+ GLUTEN_SPARK_PROFILE: 'spark-4.1'
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: '4.1'
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix.
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar | head -n 1)
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3554095748
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,446 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib artifact it already built, so the expensive
+ # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a
+ # called workflow runs as part of the caller run), so the jobs below download it
+ # by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
Review Comment:
`ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true` weakens GitHub Actions’ safety checks by allowing deprecated Node runtimes for actions. This workflow only uses v4 actions (Node20) in the shown steps, so keeping this enabled increases blast radius without an evident need.
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,747 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline file exists but is empty (not yet bootstrapped) the mode
+ automatically degrades to ``seed`` so the first run is never spuriously red.
+ A *missing* ``--known-failures`` file is treated as a configuration error
+ (the gate fails) so a mis-referenced path can't silently pass.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard). Pass ``--expected-shards N``
+ to fail when fewer than ``N`` shards contributed gate lists (a shard that
+ died before writing them), so an incomplete baseline is never produced.
+
+Flaky quarantine (``--flaky-tests``)
+ Some Delta-on-Gluten failures are non-deterministic (e.g. a native bug that
+ only triggers on certain runtime plans), so they are neither a stable pass
+ nor a stable failure and cannot live in the baseline: baselining them turns
+ the gate red on every run where they pass, and leaving them out turns it red
+ on every run where they fail. ``flaky-tests.txt`` quarantines them -- a
+ quarantined test never counts as a regression (when it fails) nor as
+ now-passing (when it passes), and is excluded from the regenerated baseline.
+ Its SUITE is an fnmatch glob (so one line covers a root-cause family across
+ generated suite variants); its TEST name is matched exactly.
+
+Flaky quarantine by error signature (``--flaky-error-patterns``)
+ When a failure is caused by a known nondeterministic bug that surfaces on a
+ *different test each run* (e.g. the native Delta DV bitmap row-index error),
+ matching by test name is whack-a-mole. ``flaky-error-patterns.txt`` instead
+ quarantines by root cause: each line is a regex matched against a failed
+ test's <failure>/<error> text, and any failure that matches is treated as
+ flaky regardless of which test it landed on.
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import fnmatch
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+
+class NoReportsError(RuntimeError):
+ """Raised when no JUnit <testsuite> elements are found under reports_dir."""
+
+
+class CorruptReportError(NoReportsError):
+ """Raised when an expected JUnit report file (TEST-*.xml) fails to parse.
+
+ Subclasses NoReportsError so the enforce/seed handler treats a truncated
+ report as a hard data error (exit 2) instead of silently dropping the
+ suite's results and letting the gate pass on partial data.
+ """
+
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def make_is_flaky(flaky_entries):
+ """Build a predicate that matches a (suite, test) tuple against flaky entries.
+
+ A flaky entry quarantines a test whose failure is known to be non-deterministic
+ (see flaky-tests.txt). The entry's SUITE is treated as an fnmatch glob so a
+ single line can cover a root-cause family across generated suite variants
+ (e.g. ``*DVs*Suite`` matches every deletion-vector merge suite, ``*`` matches
+ any suite); the TEST name is matched exactly (test names are freeform and may
+ contain glob metacharacters, so they are never globbed).
+ """
+ exact = set()
+ globbed = []
+ for suite, test in flaky_entries:
+ if any(ch in suite for ch in "*?["):
+ globbed.append((suite, test))
+ else:
+ exact.add((suite, test))
+
+ def is_flaky(entry):
+ if entry in exact:
+ return True
+ suite, test = entry
+ for glob_suite, glob_test in globbed:
+ if test == glob_test and fnmatch.fnmatchcase(suite, glob_suite):
+ return True
+ return False
+
+ return is_flaky
+
+
+def load_patterns(path):
+ """Load flaky-error regex patterns from a file (one per line).
+
+ Blank lines and ``#`` comments are ignored. Each remaining line is compiled
+ as a case-sensitive regex. These match the FAILURE TEXT of a failed test
+ (its JUnit <failure>/<error> message + stack), so a test that fails with a
+ known-nondeterministic native error (e.g. the Delta DV bitmap row-index bug)
+ can be quarantined by root cause instead of by exact test name.
+ """
+ patterns = []
+ if not path or not os.path.exists(path):
+ return patterns
+ with open(path, encoding="utf-8") as fh:
+ for line in fh:
+ line = line.rstrip("\n")
+ if not line.strip() or line.lstrip().startswith("#"):
+ continue
+ patterns.append(re.compile(line))
+ return patterns
+
+
+def make_signature_matcher(patterns):
+ """Return a predicate matching a failure text against any flaky-error pattern."""
+
+ def matches(text):
+ if not text:
+ return False
+ return any(p.search(text) for p in patterns)
+
+ return matches
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def _failure_text(tc):
+ """Concatenate the message attribute + body text of a testcase's
+ <failure>/<error> children, for error-signature matching."""
+ parts = []
+ for c in tc:
+ if c.tag.split("}")[-1] in ("failure", "error"):
+ msg = c.get("message")
+ if msg:
+ parts.append(msg)
+ if c.text:
+ parts.append(c.text)
+ return "\n".join(parts)
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped, fail_texts). The first three are sets of
+ (suite, test) tuples; fail_texts maps each failed (suite, test) to its
+ combined <failure>/<error> message + stack text (used for error-signature
+ quarantine). A test is 'failed' if its <testcase> has a <failure> or <error>
+ child, 'skipped' if it has a <skipped> child, otherwise 'passed'. Suite-level
+ aborts (a <testsuite> reporting errors/failures with no failing <testcase>)
+ are recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+ fail_texts = {}
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ # A TEST-*.xml that fails to parse is almost always a report truncated
+ # when a forked test JVM was killed mid-write (e.g. OOM). Silently
+ # skipping it drops that suite's results and could let the gate go
+ # green on partial data, so fail hard for report files. Other XML that
+ # merely matched the broad `target/**` glob is still skipped.
+ if os.path.basename(xml_file).startswith("TEST-"):
+ raise CorruptReportError(
+ "corrupt or truncated JUnit report {}: {}. Refusing to "
+ "evaluate the gate on partial data.".format(xml_file, exc)
+ )
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = (suite, name)
+ tags = _child_local_tags(tc)
+ if "failure" in tags or "error" in tags:
+ failed.add(key)
+ suite_has_failing_tc = True
+ fail_texts[key] = _failure_text(tc)
+ elif "skipped" in tags:
+ skipped.add(key)
+ else:
+ passed.add(key)
+
+ # Suite-level abort: counters say something failed but no testcase
+ # carried the failure (the suite blew up in beforeAll/constructor).
+ # Record a
+ # synthetic entry so the regression is visible.
+ try:
+ errors = int(ts.get("errors", "0") or "0")
+ failures = int(ts.get("failures", "0") or "0")
+ except ValueError:
+ errors = failures = 0
+ if (errors + failures) > 0 and not suite_has_failing_tc:
+ failed.add((suite_name, SUITE_ABORTED))
+
+ if not parsed_any:
+ raise NoReportsError(
+ "No JUnit <testsuite> elements found under {}. The test reports are "
+ "missing or in an unexpected format -- refusing to evaluate the gate "
+ "on an empty result set (this would otherwise pass silently).".format(
+ reports_dir
+ )
+ )
+
+ # A test can't be both passed and failed; failure wins. Skipped only counts
+ # if the test was not otherwise seen (e.g. retried).
+ passed -= failed
+ skipped -= failed
+ skipped -= passed
+ return passed, failed, skipped, fail_texts
+
+
+# --------------------------------------------------------------------------- #
+# Reporting helpers
+# --------------------------------------------------------------------------- #
+def _summary_sink():
+ """Return a writer that mirrors to GITHUB_STEP_SUMMARY when available."""
+ path = os.environ.get("GITHUB_STEP_SUMMARY")
+ handle = open(path, "a", encoding="utf-8") if path else None
+
+ def write(line=""):
+ print(line)
+ if handle:
+ handle.write(line + "\n")
+
+ return write, handle
+
+
+def _print_block(write, title, entries, limit=50):
+ write("")
+ write("### {} ({})".format(title, len(entries)))
+ if not entries:
+ return
+ write("")
+ write("```")
+ for i, (suite, test) in enumerate(sorted(entries)):
+ if i >= limit:
+ write("... and {} more".format(len(entries) - limit))
+ break
+ write(format_entry(suite, test))
+ write("```")
+
+
+# --------------------------------------------------------------------------- #
+# Modes
+# --------------------------------------------------------------------------- #
+def run_enforce(args):
+ # In enforce mode a missing baseline file would make load_entries() return an
+ # empty set, silently degrading to seed mode and passing the gate without
+ # enforcing anything. Treat a missing path as a configuration error; an
+ # existing-but-empty file is still allowed (it legitimately seeds).
+ if args.mode == "enforce" and (
+ not args.known_failures or not os.path.exists(args.known_failures)
+ ):
+ eprint(
+ "ERROR: --known-failures '{}' does not exist. In enforce mode the "
+ "baseline file must exist (an existing-but-empty file is allowed and "
+ "triggers seed mode). Refusing to silently pass.".format(
+ args.known_failures
+ )
+ )
+ return 2
+ baseline = load_entries(args.known_failures)
+ name_flaky = make_is_flaky(load_entries(args.flaky_tests))
+ sig_matches = make_signature_matcher(load_patterns(args.flaky_error_patterns))
+ try:
+ passed, failed, skipped, fail_texts = parse_reports(args.reports_dir)
+ except NoReportsError as exc:
+ eprint("ERROR: {}".format(exc))
+ return 2
+
+ # A test is quarantined if its NAME is in flaky-tests.txt, or its failure
+ # TEXT matches a flaky-error signature (e.g. the native DV bitmap row-index
+ # bug that hits a different DV-merge test each run).
+ def sig_flaky(e):
+ return e in fail_texts and sig_matches(fail_texts[e])
+
+ def flaky_is(e):
+ return name_flaky(e) or sig_flaky(e)
+
+ # Always emit this shard's artifacts for the aggregation job. Signature-flaky
+ # failures are dropped from failures-out: the aggregate job works off these
+ # text-less lists and cannot re-derive the signature match, so excluding them
+ # here keeps the regenerated baseline from absorbing a flaky failure. Name-
+ # flaky entries stay (the aggregate re-filters them via flaky-tests.txt).
+ if args.failures_out:
+ write_entries(args.failures_out, {e for e in failed if not sig_flaky(e)})
+ if args.ran_out:
+ write_entries(args.ran_out, passed | failed)
+
+ write, handle = _summary_sink()
+ try:
+ seeding = args.mode == "seed" or not baseline
+ if seeding and args.mode != "seed":
+ write(
+ "> NOTE: baseline `{}` is empty -- running in SEED mode "
+ "(no failures will be enforced). Bootstrap the baseline from "
+ "the aggregated artifact, commit it, then enforcement begins.".format(
+ args.known_failures
+ )
+ )
+
+ write(
+ "## Delta-on-Gluten test gate -- shard {}".format(
+ os.environ.get("SHARD_ID", "?")
+ )
+ )
+ write("")
+ write("| Category | Count |")
+ write("|---|---:|")
+ write("| Ran (pass+fail) | {} |".format(len(passed) + len(failed)))
+ write("| Passed | {} |".format(len(passed)))
+ write("| Failed | {} |".format(len(failed)))
+ write("| Skipped | {} |".format(len(skipped)))
+ write("| Baseline (known failures) | {} |".format(len(baseline)))
+
+ if seeding:
+ write("")
+ write(
+ "Seed mode: recorded {} failing test(s) for this shard. "
+ "Nothing enforced.".format(len(failed))
+ )
+ return 0
+
+ regressions = {e for e in (failed - baseline) if not flaky_is(e)}
+ quarantined = {e for e in (failed - baseline) if flaky_is(e)}
+ # `- flaky` on fixed is defensive: flaky tests are excluded from the
+ # regenerated baseline (aggregate mode), so a flaky test should never be in
+ # `baseline` in the first place -- but if one slips in, don't let its
+ # non-deterministic pass trip the now-passing gate.
+ fixed = {e for e in (baseline & passed) if not flaky_is(e)}
+ expected = failed & baseline
+
+ write("")
+ write("| Gate result | Count |")
+ write("|---|---:|")
+ write("| Expected failures (in baseline) | {} |".format(len(expected)))
+ write("| **Regressions (new failures)** | {} |".format(len(regressions)))
+ write("| Now-passing (remove from baseline) | {} |".format(len(fixed)))
+ write("| Quarantined flaky failures (ignored) | {} |".format(len(quarantined)))
+
+ _print_block(
+ write, "Regressions -- new failures NOT in the baseline", regressions
+ )
+ if regressions:
+ write("")
+ write(
+ "These tests were not previously known to fail. Either fix "
+ "the regression, or (if it is a genuinely new expected "
+ "failure) add the lines above to `known-failures.txt`."
+ )
+
+ _print_block(
+ write,
+ "Quarantined flaky failures -- ignored (flaky-tests.txt + flaky-error-patterns.txt)",
+ quarantined,
+ )
+
+ if args.fail_on_fixed:
+ _print_block(
+ write, "Now-passing -- delete these lines from the baseline", fixed
+ )
+
+ exit_code = 0
+ if regressions:
+ for suite, test in sorted(regressions):
+ eprint("::error::REGRESSION {}".format(format_entry(suite, test)))
+ exit_code = 1
+ if args.fail_on_fixed and fixed:
+ for suite, test in sorted(fixed):
+ eprint(
+ "::error::NOW-PASSING (remove from baseline) {}".format(
+ format_entry(suite, test)
+ )
+ )
+ exit_code = 1
+
+ if exit_code == 0:
+ write("")
+ write("All failures are expected (in the baseline). Gate passed.")
+ return exit_code
+ finally:
+ if handle:
+ handle.close()
+
+
+def _shard_ids(files, prefix):
+ """Return the set of shard ids from gate-list filenames.
+
+ Gate lists are named ``<prefix><shard-id>.txt`` (e.g. ``failures-shard-0.txt``,
+ ``ran-shard-0.txt``); this extracts the ``<shard-id>`` token so aggregate mode
+ can count how many shards contributed. Matching is on the basename, so nested
+ download dirs are fine.
+ """
+ ids = set()
+ pat = re.compile(r"^" + re.escape(prefix) + r"(.+)\.txt$")
+ for f in files:
+ m = pat.match(os.path.basename(f))
+ if m:
+ ids.add(m.group(1))
+ return ids
+
+
+def run_aggregate(args):
+ failure_files = sorted(
+ glob.glob(os.path.join(args.inputs_dir, "**", "failures-*.txt"), recursive=True)
+ )
+ ran_files = sorted(
+ glob.glob(os.path.join(args.inputs_dir, "**", "ran-*.txt"), recursive=True)
+ )
+
+ # No per-shard gate lists means the artifacts were never produced or the
+ # download failed (the workflow's download step is continue-on-error). Bail
+ # out before writing an empty baseline-out, which could otherwise be committed
+ # and wipe the entire known-failures.txt.
+ if not failure_files and not ran_files:
+ eprint(
+ "ERROR: no per-shard failures-*.txt / ran-*.txt files found under "
+ "{}. Refusing to aggregate an empty baseline (gate-list artifacts are "
+ "missing or were not downloaded).".format(args.inputs_dir)
+ )
+ return 2
+
+ # Completeness guard: each shard writes its failures-<id>.txt and ran-<id>.txt
+ # together (see run_enforce), so a shard that died before the gate step -- or
+ # whose artifact failed to download -- contributes neither. The empty-inputs
+ # check above only catches losing *all* shards; without this a partial set
+ # would silently regenerate a baseline missing that shard's failures, wrongly
+ # shrinking known-failures.txt and reddening the next run. A shard counts as
+ # complete only when BOTH its files are present (robust to partial downloads).
+ if args.expected_shards:
+ complete = _shard_ids(failure_files, "failures-") & _shard_ids(
+ ran_files, "ran-"
+ )
+ if len(complete) != args.expected_shards:
+ eprint(
+ "ERROR: expected {} shard gate-list set(s) but found {} complete "
+ "(shards: {}). A shard's failures-*/ran-* artifact is missing -- it "
+ "likely died before writing its gate lists or its artifact failed "
+ "to download -- so the regenerated baseline would be incomplete and "
+ "could wrongly shrink known-failures.txt. Refusing to aggregate.".format(
+ args.expected_shards,
+ len(complete),
+ ", ".join(sorted(complete)) or "none",
+ )
+ )
+ return 2
+
+ union_failed = set()
+ for f in failure_files:
+ union_failed |= load_entries(f)
+ union_ran = set()
+ for f in ran_files:
+ union_ran |= load_entries(f)
+
+ flaky_is = make_is_flaky(load_entries(args.flaky_tests))
+ # Exclude quarantined flaky tests from the regenerated baseline: a flaky test
+ # that happened to fail this run must never be baked into known-failures.txt
+ # (otherwise it would trip the now-passing gate on the next run where it
+ # passes). Flaky failures are tracked in flaky-tests.txt, not the baseline.
+ baseline_body = {e for e in union_failed if not flaky_is(e)}
+
+ header = (
+ "# Known Delta-on-Gluten unit test failures.\n"
+ "#\n"
+ "# Auto-generated by compare-test-results.py --mode aggregate.\n"
+ "# Format: <fully.qualified.SuiteName>#<test display name>\n"
+ "# Lines starting with '#' are comments.\n"
+ "#\n"
+ "# Regenerate by running the 'Delta Spark UT (Gluten)' workflow with\n"
+ "# update_baseline=true and committing the produced artifact.\n"
+ )
Review Comment:
The aggregate baseline header is what contributors are told to commit (via README/workflow), but it omits the important “regenerate under the same shard x fork parallelism” warning that exists in the committed known-failures header. If someone replaces known-failures.txt with the artifact, they’ll lose that guidance and can accidentally capture an incompatible baseline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3532695569
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,509 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline file exists but is empty (not yet bootstrapped) the mode
+ automatically degrades to ``seed`` so the first run is never spuriously red.
+ A *missing* ``--known-failures`` file is treated as a configuration error
+ (the gate fails) so a mis-referenced path can't silently pass.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard).
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import glob
+import os
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+
+class NoReportsError(RuntimeError):
+ """Raised when no JUnit <testsuite> elements are found under reports_dir."""
+
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped) sets of (suite, test) tuples. A test is
+ 'failed' if its <testcase> has a <failure> or <error> child, 'skipped' if
+ it has a <skipped> child, otherwise 'passed'. Suite-level aborts (a
+ <testsuite> reporting errors/failures with no failing <testcase>) are
+ recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
Review Comment:
a TEST-*.xml that fails to parse now raises CorruptReportError
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3617361437
##########
.github/workflows/util/delta-spark-ut/run-delta-tests.sh:
##########
@@ -0,0 +1,240 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Runs the Delta `spark` module tests for one shard under the Gluten bundle:
+# arms a hang watchdog (thread-dumps + kills a wedged fork), invokes sbt
+# spark/test with the tuned JVM/heap flags, prints cgroup memory forensics, and
+# then gates the results against the baseline (compare-test-results.py). Extracted
+# from delta_spark_ut.yml so the workflow step stays readable.
+#
+# Driven by environment (set by the workflow step / job):
+# SHARD_ID - this shard's id (matrix.shard)
+# SPARK_VERSION - Delta -DsparkVersion value
+# UPDATE_BASELINE - 'true' -> gate seed mode; else enforce
+# FAIL_ON_FIXED - passed through to the gate
+# DELTA_SCALA_VERSION, NUM_SHARDS, TEST_PARALLELISM_COUNT, DELTA_TESTING
+# - test env (see the workflow step's `env:` block)
+# GITHUB_WORKSPACE - repo root (holds the Delta clone + util scripts)
+#
+# JAVA_TOOL_OPTIONS is set by sourcing java-test-args.sh (below), not the caller.
+
+set -euo pipefail
+export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+export PATH=$JAVA_HOME/bin:$PATH
+# Gluten/JDK17 test JVM flags (--add-opens + Netty property), shared with local
+# dev runs. Sets JAVA_TOOL_OPTIONS so it reaches the sbt launcher + forked JVMs.
+# shellcheck source=./java-test-args.sh
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/java-test-args.sh"
+cd "$GITHUB_WORKSPACE/delta"
+chmod +x build/sbt
+# Only run the unified `spark` sbt project, NOT `sparkGroup/test` --
+# `sparkGroup` aggregates many other projects (sparkV2, contribs,
+# sharing, connect*, ...) that are out of scope for this pipeline.
+#
+# JVM heap layout -- two memory consumers on the ~16G runner:
+# * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to
+# return idle memory during the (long) test phase via G1 periodic GC
+# (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the
+# busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent
+# forces each periodic GC to a full STW collection) that uncommits to a
+# tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low
+# -Xms512m floor.
+# Without this the idle launcher holds ~5.3G for the whole run; with
+# it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime
+# config, so they cannot affect the measured pass/fail signal.
+# * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command
+# below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends
+# so our -Xmx2G comes last and wins. Gluten offloads data to Velox
+# off-heap (capped at 2g via spark.memory.offHeap.size in the patched
+# DeltaSQLCommandTest), so the fork's heap need is modest. A larger
+# fork heap pushed the cgroup peak past the ~16G OOM threshold and the
+# kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G
+# keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is
+# analyzable.
+# `-u target/test-reports` enables ScalaTest's JUnit XML reporter so
+# every suite writes per-test results. Delta itself only configures
+# the console reporter (-oDF), so without this we'd have no machine-
+# readable results to gate on. The path is relative to the forked
+# test JVM's working dir (Test / baseDirectory = spark/), i.e.
+# delta/spark/target/test-reports/TEST-*.xml.
+#
+# We deliberately do NOT let an sbt non-zero exit (which fires on the
+# MANY expected Delta-on-Gluten failures) fail this step directly.
+# Instead the known-failures gate below decides pass/fail: the build
+# is green when the only failures are ones already recorded in the
+# baseline, and red on a genuine regression.
+set +e
+# --- hang watchdog ---------------------------------------------------
+# Shard 2 (and occasionally others) hangs indefinitely after a suite's
+# last test with no further output. ScalaTest's failAfter only wraps
+# individual test BODIES, so a wedge in suite teardown/afterAll -- or in
+# a non-interruptible native Velox/JNI call that ignores
+# Thread.interrupt() -- has no timeout and stalls until the 350-min job
+# limit with zero diagnostics. This watchdog dumps the forked test JVM's
+# threads (to the job log, and to a file for the artifact) once the test
+# output has been silent for too long, so the deadlock is diagnosable.
+SBT_LOG="/tmp/sbt-spark-test-shard-${SHARD_ID}.log"
+: > "$SBT_LOG"
+rm -f /tmp/sbt-done
+(
+ # CRITICAL: the step shell runs with `bash -eo pipefail`, which the
+ # subshell inherits. Without `set +e` here, ANY non-zero command --
+ # e.g. fork detection finding no match, or `kill`/`jps` returning
+ # non-zero -- silently kills this watchdog. That errexit kill (plus a
+ # /proc detection miss) once made the watchdog capture ZERO dumps. A
+ # diagnostic must never abort on a failed probe.
+ set +e +o pipefail
+ JSTACK="${JAVA_HOME}/bin/jstack"
+ JPS="${JAVA_HOME}/bin/jps"
+ silent_limit=900 # 15 min with no new test output => treat as hung
+ dumps=0
+ fork_pids() {
+ # The sbt test fork's main class is sbt.ForkMain. Prefer jps (reads
+ # the main class from hsperfdata, robust to sbt's @argfile launch);
+ # fall back to scanning /proc cmdline + @argfile.
+ "$JPS" -l 2>/dev/null | awk '/sbt\.ForkMain/ {print $1}'
+ local p cl arg
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] || continue
+ cl="$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)"
+ case "$cl" in *sbt.ForkMain*) echo "${p##*/}"; continue ;; esac
+ arg="$(printf '%s' "$cl" | tr ' ' '\n' | sed -n 's/^@//p' | head -1)"
+ [ -n "$arg" ] && [ -f "$arg" ] && grep -qa 'sbt\.ForkMain' "$arg" 2>/dev/null \
+ && echo "${p##*/}"
+ done
+ }
+ all_java_pids() {
+ "$JPS" -q 2>/dev/null
+ local p
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] && echo "${p##*/}"
+ done
+ }
+ echo "HANG WATCHDOG armed: dumps the test JVM after ${silent_limit}s of output silence"
+ hb=0
+ while [ ! -f /tmp/sbt-done ]; do
+ sleep 60
+ [ -f "$SBT_LOG" ] || continue
+ now=$(date +%s)
+ mtime=$(stat -c %Y "$SBT_LOG" 2>/dev/null || echo "$now")
+ silent=$(( now - mtime ))
+ # Per-minute memory profile: heap tuning proved the ~16G OOM peak is
+ # NATIVE-driven, so log which JVM (sbt launcher vs fork) actually grows
+ # toward it -- the last lines before a hang reveal the real hog to cut.
+ # Read /proc directly (no `ps` dependency in the minimal container).
+ memnow=$(awk '{printf "%.2fG",$1/1073741824}' /sys/fs/cgroup/memory.current 2>/dev/null)
+ jvmrss=""
+ for mp in $(all_java_pids 2>/dev/null | sort -un); do
+ r=$(awk '/^VmRSS:/{print $2}' "/proc/$mp/status" 2>/dev/null)
+ [ -n "$r" ] && jvmrss="$jvmrss $(( r / 1024 ))M(p$mp)"
+ done
+ echo "MEM cgroup=${memnow} JVMs=[${jvmrss# }]"
+ hb=$(( hb + 1 ))
+ # Heartbeat every ~5 min so we can SEE the watchdog is alive (and how
+ # long the test has been silent) without waiting for a hang.
+ [ $(( hb % 5 )) -eq 0 ] && echo "HANG WATCHDOG: alive; last test output ${silent}s ago"
+ if [ "$silent" -ge "$silent_limit" ] && [ "$dumps" -lt 3 ]; then
+ dumps=$(( dumps + 1 ))
+ pids="$(fork_pids | sort -un)"
+ # Safety net: if the fork JVM cannot be pinpointed, dump EVERY JVM.
+ [ -n "$pids" ] || pids="$(all_java_pids | sort -un)"
+ echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $pids))"
+ [ -n "$pids" ] || echo "HANG WATCHDOG: no java process found to dump"
Review Comment:
Agreed. I split the dump set from the kill set: we still dump *every* JVM for diagnostics when no `sbt.ForkMain` fork can be pinpointed, but we now only KILL matched fork pids. During a pre-fork stall (dependency resolution or a cold-cache compile) the watchdog dumps and leaves sbt running instead of killing the launcher, so we don't waste the slot on a confusing "compile/launch failure". Only a real wedged fork is killed (and only then is the fail-the-shard marker set). I also made the per-episode dump/kill budget reset when output resumes, so a transient pre-fork stall we dump-but-don't-kill can't starve the budget for a real fork hang later. (73e5c90)
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,747 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline file exists but is empty (not yet bootstrapped) the mode
+ automatically degrades to ``seed`` so the first run is never spuriously red.
+ A *missing* ``--known-failures`` file is treated as a configuration error
+ (the gate fails) so a mis-referenced path can't silently pass.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard). Pass ``--expected-shards N``
+ to fail when fewer than ``N`` shards contributed gate lists (a shard that
+ died before writing them), so an incomplete baseline is never produced.
+
+Flaky quarantine (``--flaky-tests``)
+ Some Delta-on-Gluten failures are non-deterministic (e.g. a native bug that
+ only triggers on certain runtime plans), so they are neither a stable pass
+ nor a stable failure and cannot live in the baseline: baselining them turns
+ the gate red on every run where they pass, and leaving them out turns it red
+ on every run where they fail. ``flaky-tests.txt`` quarantines them -- a
+ quarantined test never counts as a regression (when it fails) nor as
+ now-passing (when it passes), and is excluded from the regenerated baseline.
+ Its SUITE is an fnmatch glob (so one line covers a root-cause family across
+ generated suite variants); its TEST name is matched exactly.
+
+Flaky quarantine by error signature (``--flaky-error-patterns``)
+ When a failure is caused by a known nondeterministic bug that surfaces on a
+ *different test each run* (e.g. the native Delta DV bitmap row-index error),
+ matching by test name is whack-a-mole. ``flaky-error-patterns.txt`` instead
+ quarantines by root cause: each line is a regex matched against a failed
+ test's <failure>/<error> text, and any failure that matches is treated as
+ flaky regardless of which test it landed on.
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import fnmatch
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+
+class NoReportsError(RuntimeError):
+ """Raised when no JUnit <testsuite> elements are found under reports_dir."""
+
+
+class CorruptReportError(NoReportsError):
+ """Raised when an expected JUnit report file (TEST-*.xml) fails to parse.
+
+ Subclasses NoReportsError so the enforce/seed handler treats a truncated
+ report as a hard data error (exit 2) instead of silently dropping the
+ suite's results and letting the gate pass on partial data.
+ """
+
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def make_is_flaky(flaky_entries):
+ """Build a predicate that matches a (suite, test) tuple against flaky entries.
+
+ A flaky entry quarantines a test whose failure is known to be non-deterministic
+ (see flaky-tests.txt). The entry's SUITE is treated as an fnmatch glob so a
+ single line can cover a root-cause family across generated suite variants
+ (e.g. ``*DVs*Suite`` matches every deletion-vector merge suite, ``*`` matches
+ any suite); the TEST name is matched exactly (test names are freeform and may
+ contain glob metacharacters, so they are never globbed).
+ """
+ exact = set()
+ globbed = []
+ for suite, test in flaky_entries:
+ if any(ch in suite for ch in "*?["):
+ globbed.append((suite, test))
+ else:
+ exact.add((suite, test))
+
+ def is_flaky(entry):
+ if entry in exact:
+ return True
+ suite, test = entry
+ for glob_suite, glob_test in globbed:
+ if test == glob_test and fnmatch.fnmatchcase(suite, glob_suite):
+ return True
+ return False
+
+ return is_flaky
+
+
+def load_patterns(path):
+ """Load flaky-error regex patterns from a file (one per line).
+
+ Blank lines and ``#`` comments are ignored. Each remaining line is compiled
+ as a case-sensitive regex. These match the FAILURE TEXT of a failed test
+ (its JUnit <failure>/<error> message + stack), so a test that fails with a
+ known-nondeterministic native error (e.g. the Delta DV bitmap row-index bug)
+ can be quarantined by root cause instead of by exact test name.
+ """
+ patterns = []
+ if not path or not os.path.exists(path):
+ return patterns
+ with open(path, encoding="utf-8") as fh:
+ for line in fh:
+ line = line.rstrip("\n")
+ if not line.strip() or line.lstrip().startswith("#"):
+ continue
+ patterns.append(re.compile(line))
+ return patterns
+
+
+def make_signature_matcher(patterns):
+ """Return a predicate matching a failure text against any flaky-error pattern."""
+
+ def matches(text):
+ if not text:
+ return False
+ return any(p.search(text) for p in patterns)
+
+ return matches
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def _failure_text(tc):
+ """Concatenate the message attribute + body text of a testcase's
+ <failure>/<error> children, for error-signature matching."""
+ parts = []
+ for c in tc:
+ if c.tag.split("}")[-1] in ("failure", "error"):
+ msg = c.get("message")
+ if msg:
+ parts.append(msg)
+ if c.text:
+ parts.append(c.text)
+ return "\n".join(parts)
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped, fail_texts). The first three are sets of
+ (suite, test) tuples; fail_texts maps each failed (suite, test) to its
+ combined <failure>/<error> message + stack text (used for error-signature
+ quarantine). A test is 'failed' if its <testcase> has a <failure> or <error>
+ child, 'skipped' if it has a <skipped> child, otherwise 'passed'. Suite-level
+ aborts (a <testsuite> reporting errors/failures with no failing <testcase>)
+ are recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+ fail_texts = {}
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ # A TEST-*.xml that fails to parse is almost always a report truncated
+ # when a forked test JVM was killed mid-write (e.g. OOM). Silently
+ # skipping it drops that suite's results and could let the gate go
+ # green on partial data, so fail hard for report files. Other XML that
+ # merely matched the broad `target/**` glob is still skipped.
+ if os.path.basename(xml_file).startswith("TEST-"):
+ raise CorruptReportError(
+ "corrupt or truncated JUnit report {}: {}. Refusing to "
+ "evaluate the gate on partial data.".format(xml_file, exc)
+ )
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = (suite, name)
Review Comment:
Nice one — subtle but real. Added a `normalize_key(suite, test)` helper that runs the XML-derived key through the same round-trip the baseline uses (`write_entries` collapses CR/LF in the test name to spaces; `parse_entry` strips the whole `suite#test` line), applied at both key sites (the testcase key and the suite-abort key). It's a no-op for normal names, but now a name with a trailing newline/whitespace normalizes to match a line pasted from the gate's `REGRESSION` output, so it's actually suppressible. Verified with an enforce-mode run using a report whose testcase `name` carries a trailing newline. (73e5c90)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3748429977
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,469 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Self-contained: this workflow owns the whole Delta pipeline, including its own
+ # native build, and is NOT called by velox_backend_x86.yml. A change confined to
+ # this pipeline's own files no longer drags in that workflow's ~50-job
+ # TPC-H/DS + Spark-UT matrix (measured: ~1750 runner-minutes per run that such a
+ # change cannot affect), and a Velox/core change no longer carries the Delta
+ # suite.
+ #
+ # Note the tradeoff: `gluten-delta/**` and `backends-velox/src-delta*/**` also
+ # match velox_backend_x86.yml's own filter (its spark-ut jobs build with
+ # -Pdelta), so a change there runs BOTH workflows and, now that the native lib
+ # is no longer shared between them, pays for the centos-7 build twice (~10 min).
+ # That is the price of decoupling, and it is small next to the case above.
+ #
+ # The `paths:` filter below IS the per-PR gate -- GitHub evaluates it before the
+ # run is created, so a non-Delta PR costs nothing at all. Changes to general
+ # Velox/core/native code can also affect Delta offload, but they are touched on
+ # most PRs; the nightly `schedule` run below is the safety net for those, and
+ # `workflow_dispatch` lets anyone run the suite against a branch on demand.
+ pull_request:
+ paths:
+ - '.github/workflows/delta_spark_ut.yml'
+ - '.github/workflows/util/delta-spark-ut/**'
+ - 'gluten-delta/**'
+ # Covers src-delta, src-delta33, src-delta40 and any future variant.
+ - 'backends-velox/src-delta*/**'
Review Comment:
I assume changes to Gluten's common code may also affect the Delta support. How do we prevent such regression in the future?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3668000437
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -101,6 +106,80 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR
+ # touches high-signal Delta paths -- the Delta integration code
+ # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own
+ # files -- or carries the `run-delta-ci` opt-in label. Changes to general
+ # Velox/core/native code can also affect Delta offload but are touched
+ # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml
+ # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage
+ # down. NOTE: the label is read from the event that triggered this run, so add
+ # it before/with a push; labeling an already-finished run needs a new push.
+ delta-changes:
+ runs-on: ubuntu-22.04
+ outputs:
+ run_delta: ${{ steps.filter.outputs.run_delta }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ - name: Detect Delta-relevant changes / opt-in label
+ id: filter
+ env:
+ HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+ run: |
+ set -euo pipefail
+ # Opt-in label forces the suite even with no Delta-relevant path change.
+ if [ "$HAS_LABEL" = "true" ]; then
+ echo "run-delta-ci label present -> running Delta suite"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
+ fi
+ # Fail open if we can't determine the PR range (e.g. a non-PR trigger):
+ # never silently skip coverage.
+ if [ -z "${BASE_SHA:-}" ] || [ -z "${HEAD_SHA:-}" ]; then
+ echo "no PR base/head sha -> running Delta suite (fail-open)"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
+ fi
+ BASE=$(git merge-base "$BASE_SHA" "$HEAD_SHA" 2>/dev/null || echo "$BASE_SHA")
+ echo "diff base=$BASE head=$HEAD_SHA"
+ # Fail open if the diff itself can't be computed (missing objects after a
+ # force-push race, an unfetched fork head, ...). Piping straight into
+ # `grep -q` inside an `if` would hide that: git's failure leaves grep with
+ # empty input, so the pipeline exits non-zero exactly as it does for "no
+ # match" -- and `set -e`/`pipefail` can't help, since a tested command is
+ # allowed to fail. Capture the diff first so the two cases stay distinct.
+ if ! CHANGED=$(git diff --name-only "$BASE" "$HEAD_SHA"); then
Review Comment:
The PR diff detection relies on `github.event.pull_request.head.sha`, which is frequently not present in the local checkout for fork PRs (checkout typically fetches the PR merge ref in the base repo, not the fork head SHA). In that case `git diff ... \"$HEAD_SHA\"` will fail and the logic intentionally ‘fail-opens’ to running the expensive Delta suite—effectively disabling the cost-saving gate for fork PRs. Consider diffing against the checked-out commit (`github.sha`, which is available) instead of the fork head SHA, e.g. compare `BASE_SHA` (or merge-base) to the merge commit that Actions checked out. This keeps the gate effective while retaining the existing fail-open behavior for genuine diff computation failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5126211935
@felipepessoto, thanks for the update. The new delta-changes gate nicely protects the expensive Delta suite from unrelated changes. One asymmetry worth noting: the reverse isn't gated, a change confined to gluten-delta/** or the Delta pipeline files still triggers the full TPC-H/DS + Spark-UT matrix, since those jobs have no path filter. I am wondering whether we can optimize the CI workflow further.
Could we use Apache Stash to cache the native lib so it can be shared across workflows? That would let us move the Delta test jobs into a separate workflow file. cc @zhouyuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5111229987
@philo-he, needs your help to merge it. It will help with a couple of open Delta PRs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5015174080
> > It seems that, rather than fixing TIMESTAMP_NTZ itself, we may need to make Gluten Delta handle fallbacks correctly. What do you think?
>
> Yes, unsupported operations should also induce fallback for Delta plans. @felipepessoto would you file an issue accordingly?
>
> By the way, will the new tests introduced in this PR overlap the existing ones in Gluten's test codebase? Can we remove the overlapped ones?
>
> cc @malinjawi
@zhztheplayer @felipepessoto Sorry for the delay. I went through our test tree to give you a concrete answer.
Yes there's overlap, but it's smaller than it looks. The overlapped ones are the 8 suite files we imported from delta-io/delta over time (`DeltaSuite`, `DeleteSQLSuite` and its DV variants, `UpdateSQLSuite`, `DeltaDDLSuite`, `DeltaInsertIntoTableSuite`,`ClusteredTableClusteringSuite`, `OptimizeMetadataOnlyDeltaQuerySuite`, `OptimizedWritesSuite`). And even those are only redundant on the Spark 4.1 leg, because this pipeline runs Delta 4.2 + Spark 4.1 only. Today those copies are the only Delta tests we have on Spark 3.5 and 4.0. If we delete them now, the older Spark lines lose Delta coverage completely.
Everything else is not overlap. `DeltaNativeWriteSuite`, `DeltaDeletionVectorHandoffSuite`, `DeltaDeletionVectorScanInfoSuite`, `DeltaVariantWriteSuite`, the plan-assertion tests in gluten-delta and `VeloxTPCHDeltaSuite` all check Gluten behavior (offload, fallback, DV handoff) that upstream's suites simply don't test. Those stay no matter what.
One caveat before we ever delete the copies: they aren't verbatim imports. They carry a lot of triage knowledge, like the `DeltaExcludedTestMixin` exclusion lists with per-test reasons, and assertions changed to `intercept[Exception with SparkThrowable]` because exceptions that cross the JNI boundary come back as different types. If we drop the files
without porting those notes into known-failures.txt comments, we lose the "why" behind each divergence.
So I'm with @felipepessoto : keep everything until this has baked, then remove in stages. The Spark 4.1 exposure of the src-delta40 copies is the natural first candidate, and we can go further if the pipeline ever grows a 3.5/4.0 leg (though his shard-cost point is fair).
I'm happy to do the removal pass when the time comes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3617914543
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -94,6 +99,70 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Delta Spark UT, run via the reusable workflow so it reuses the native lib
+ # built above instead of duplicating the native build. Not gated on Delta-only
+ # paths: core/velox/substrait/cpp/shims changes can affect Delta query offload,
+ # so this runs on every trigger like the other spark-test jobs.
Review Comment:
Good catch — fixed in c94aeed3e. Removed the stale comment block (it pre-dated the gating and still said the suite "runs on every trigger"); the `delta-changes` job keeps the accurate per-PR gating comment, and the `delta-spark-ut` job now has a short note that it reuses the native lib and is gated by `delta-changes` (nightly full run lives in delta_spark_ut.yml's `schedule`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-4989409743
> It seems that, rather than fixing TIMESTAMP_NTZ itself, we may need to make Gluten Delta handle fallbacks correctly. What do you think?
Yes, unsupported operations should also induce fallback for Delta plans. @felipepessoto would you file an issue accordingly?
By the way, will the new tests introduced in this PR overlap the existing ones in Gluten's test codebase? Can we remove the overlapped ones?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3487529897
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,206 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# Shallow clone the requested tag/branch. Fall back to full clone when the ref is a SHA.
+if ! git clone --depth 1 --branch "$DELTA_REF" https://github.com/delta-io/delta.git "$DELTA_DIR"; then
+ echo "Shallow clone of ref '${DELTA_REF}' failed, falling back to full clone."
+ rm -rf "$DELTA_DIR"
+ git clone https://github.com/delta-io/delta.git "$DELTA_DIR"
+ git -C "$DELTA_DIR" checkout "$DELTA_REF"
+fi
+git -C "$DELTA_DIR" --no-pager log -1 --oneline
+echo "::endgroup::"
+
+echo "::group::Injecting Gluten bundle jar onto the spark project's TEST classpath"
+# The Gluten bundle jar must be on the spark project's TEST runtime classpath
+# (so DeltaSQLCommandTest can load org.apache.gluten.GlutenPlugin by name) but
+# NOT on the COMPILE classpath of `sparkV1`, which is the project that holds
+# Delta's main sources. The bundle's transitive contents include extra symbols
+# under `org.apache.spark.sql` that collide with Delta's main sources -- e.g.
+# MergeOutputGeneration.scala imports both `org.apache.spark.sql._` and
+# `org.apache.spark.sql.delta.ClassicColumnConversions._`, and would then fail
+# with `reference to expression is ambiguous`.
+#
+# sbt auto-scans `<baseDirectory>/lib` via `unmanagedBase`. Two relevant
+# projects in Delta v4.2.0 have a `lib/` baseDirectory:
+# - sparkV1: `project in file("spark")` -> spark/lib
+# - spark : `project in file("spark-unified")` -> spark-unified/lib
+# unmanagedJars are project-scoped (NOT inherited by dependents), so dropping
+# the bundle into spark-unified/lib/ adds it to the unified `spark` project's
+# Compile *and* Test classpaths -- but NOT to sparkV1's. That's exactly what
+# we want:
+# * sparkV1/Compile sees ONLY Delta's regular deps -> Delta main compiles.
+# * spark/Test/fullClasspath sees the bundle -> tests load GlutenPlugin.
+# (Verified empirically: with bundle only in spark-unified/lib/, sbt's
+# `show sparkV1/Compile/dependencyClasspath` excludes the bundle and
+# `show spark/Test/fullClasspath` includes it.)
+#
+# We deliberately do NOT also drop the bundle into spark/lib/, which is what
+# caused the previous compile failure: spark/lib/ is sparkV1's unmanagedBase,
+# and putting the bundle there would re-introduce the ambiguity errors.
+SPARK_UNIFIED_LIB="$DELTA_DIR/spark-unified/lib"
+mkdir -p "$SPARK_UNIFIED_LIB"
+cp "$GLUTEN_BUNDLE_JAR" "$SPARK_UNIFIED_LIB/gluten-velox-bundle.jar"
+ls -lh "$SPARK_UNIFIED_LIB"
+echo "::endgroup::"
+
+echo "::group::Patching DeltaSQLCommandTest to enable Gluten plugin"
+TARGET="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$TARGET" ]; then
+ echo "Expected file not found in Delta clone: $TARGET" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
+ exit 1
+fi
+cp "$PATCH_SOURCE" "$TARGET"
+echo "Patched $TARGET"
+echo "--- diff vs. upstream ---"
+git -C "$DELTA_DIR" --no-pager diff -- "spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala" || true
+echo "::endgroup::"
+
+# Delta's tests collect file-source scans by matching the concrete
+# `FileSourceScanExec` case class; Gluten offloads the scan to
+# DeltaScanTransformer, a `FileSourceScanLike` sibling, so those matches miss
+# (`scala.MatchError: List()`, empty partition filters, broken column-pruning /
+# scan-metric checks across many suites). delta-io/delta#7104 and #7105 widen the
+# matches to the shared `FileSourceScanLike` interface that both the vanilla and
+# Gluten scans implement (behavior-preserving for vanilla). Both are merged
+# upstream but land after the pinned DELTA_REF (v4.2.0), so apply them here; once
+# DELTA_REF includes a commit its cherry-pick is a clean no-op and the call can go.
+#
+# Depth-2 fetch brings each fix commit and its parent, which cherry-pick needs to
+# diff against (a depth-1 fetch grafts the parent away); `-n` stages the change
+# without requiring a committer identity.
+cherry_pick_delta_fix() {
+ local sha="$1" pr="$2"
+ echo "Cherry-picking delta-io/delta${pr}"
+ git -C "$DELTA_DIR" fetch --quiet --depth 2 origin "$sha"
+ git -C "$DELTA_DIR" cherry-pick -n "$sha"
+}
Review Comment:
`git cherry-pick` is not a clean no-op when the target ref already contains (or has already backported) the change — it can fail with an "empty cherry-pick" and leave the repo in CHERRY_PICK_HEAD state, breaking setup when `delta_ref` is bumped. Add a guard to skip cherry-picking commits that are already present in the checked-out Delta ref (at minimum when the SHA is an ancestor).
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,672 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Delta `-DsparkVersion` value (must match the Gluten -P profile below)'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. Spark 4.1 + Scala 2.13 + JDK 17 matches Delta v4.2.0's
+ # default Spark version (4.1.0) from project/CrossSparkVersions.scala.
+ GLUTEN_SPARK_PROFILE: 'spark-4.1'
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: '4.1'
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
Review Comment:
The Maven cache restore runs *after* downloading the custom Arrow jars. `actions/cache` restore can overwrite files under `/root/.m2/repository`, which risks replacing the caller-provided Arrow artifacts with whatever is in the cache (defeating the purpose of passing the Arrow jars artifact). Reorder these steps so the cache is restored first, then download Arrow jars afterwards to guarantee the custom jars win.
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,467 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline is empty (not yet bootstrapped) the mode automatically
+ degrades to ``seed`` so the first run is never spuriously red.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard).
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import glob
+import os
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped) sets of (suite, test) tuples. A test is
+ 'failed' if its <testcase> has a <failure> or <error> child, 'skipped' if
+ it has a <skipped> child, otherwise 'passed'. Suite-level aborts (a
+ <testsuite> reporting errors/failures with no failing <testcase>) are
+ recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = (suite, name)
+ tags = _child_local_tags(tc)
+ if "failure" in tags or "error" in tags:
+ failed.add(key)
+ suite_has_failing_tc = True
+ elif "skipped" in tags:
+ skipped.add(key)
+ else:
+ passed.add(key)
+
+ # Suite-level abort: counters say something failed but no testcase
+ # carried the failure (the suite blew up in beforeAll/constructor).
+ # Record a
+ # synthetic entry so the regression is visible.
+ try:
+ errors = int(ts.get("errors", "0") or "0")
+ failures = int(ts.get("failures", "0") or "0")
+ except ValueError:
+ errors = failures = 0
+ if (errors + failures) > 0 and not suite_has_failing_tc:
+ failed.add((suite_name, SUITE_ABORTED))
+
+ if not parsed_any:
+ eprint(
+ "WARNING: no JUnit <testsuite> elements found under {}".format(reports_dir)
+ )
+
+ # A test can't be both passed and failed; failure wins. Skipped only counts
+ # if the test was not otherwise seen (e.g. retried).
+ passed -= failed
+ skipped -= failed
+ skipped -= passed
+ return passed, failed, skipped
+
+
+# --------------------------------------------------------------------------- #
+# Reporting helpers
+# --------------------------------------------------------------------------- #
+def _summary_sink():
+ """Return a writer that mirrors to GITHUB_STEP_SUMMARY when available."""
+ path = os.environ.get("GITHUB_STEP_SUMMARY")
+ handle = open(path, "a", encoding="utf-8") if path else None
+
+ def write(line=""):
+ print(line)
+ if handle:
+ handle.write(line + "\n")
+
+ return write, handle
+
+
+def _print_block(write, title, entries, limit=50):
+ write("")
+ write("### {} ({})".format(title, len(entries)))
+ if not entries:
+ return
+ write("")
+ write("```")
+ for i, (suite, test) in enumerate(sorted(entries)):
+ if i >= limit:
+ write("... and {} more".format(len(entries) - limit))
+ break
+ write(format_entry(suite, test))
+ write("```")
+
+
+# --------------------------------------------------------------------------- #
+# Modes
+# --------------------------------------------------------------------------- #
+def run_enforce(args):
+ baseline = load_entries(args.known_failures)
+ passed, failed, skipped = parse_reports(args.reports_dir)
+
+ # Always emit this shard's artifacts for the aggregation job.
+ if args.failures_out:
+ write_entries(args.failures_out, failed)
+ if args.ran_out:
+ write_entries(args.ran_out, passed | failed)
Review Comment:
The per-shard `ran-out` file excludes skipped tests (`passed | failed`). In aggregate mode, `stale = baseline - union_ran` will then incorrectly mark baseline tests as "stale" when they were actually executed and reported as skipped in JUnit XML. If you want stale to mean "suite/test no longer exists", consider emitting a skipped list too (or including skipped in ran-out and adjusting the aggregate "fixed" computation to avoid treating skipped as passing).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3531284466
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,682 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Delta `-DsparkVersion` value (must match the Gluten -P profile below)'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. Spark 4.1 + Scala 2.13 + JDK 17 matches Delta v4.2.0's
+ # default Spark version (4.1.0) from project/CrossSparkVersions.scala.
+ GLUTEN_SPARK_PROFILE: 'spark-4.1'
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: '4.1'
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Every input has a default (workflow_call + workflow_dispatch), so they
+ # are always set; just surface them as step outputs for the steps below.
+ env:
+ DELTA_REF: ${{ inputs.delta_ref }}
+ SPARK_VERSION: ${{ inputs.spark_version }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline }}
+ FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
+
+ - name: Download Gluten bundle jar
+ uses: actions/download-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: gluten-bundle
+
+ - name: Install minimal tools
+ run: |
+ set -euo pipefail
+ # apache/gluten:centos-9-jdk17 already has java-17, git, tar, a POSIX
+ # shell, and curl-minimal (which provides the `curl` command sbt's
+ # launcher needs). Install the rest of what Delta's build/sbt and the
+ # tests may need. We deliberately do NOT install the full `curl`
+ # package -- it conflicts with the pre-installed curl-minimal.
+ yum install -y java-17-openjdk-devel which findutils gzip python3
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ git --version
+ curl --version | head -n 1
+
+ - name: Cache sbt / Ivy / Coursier
+ uses: actions/cache@v4
+ with:
+ path: |
+ /root/.sbt
+ /root/.ivy2
+ /root/.cache/coursier
+ # Intentionally NOT keyed by ${{ matrix.shard }} -- all shards share
+ # the same dependency tree, so a single shared cache (with parallel
+ # save races resolved by GH on a first-write-wins basis) gives the
+ # best storage / hit-rate tradeoff.
+ key: delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-${{ env.DELTA_SCALA_VERSION }}
+ restore-keys: |
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-
+
+ - name: Clone and patch Delta
+ run: |
+ set -euo pipefail
+ # `2>/dev/null ... || true` keeps a no-match `ls` from aborting the step
+ # under `set -o pipefail`, so the explicit check below emits a clear
+ # error instead of a generic "cannot access".
+ GLUTEN_JAR=$(ls "$GITHUB_WORKSPACE"/gluten-bundle/gluten-velox-bundle-spark*_*-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$GLUTEN_JAR" ] || [ ! -f "$GLUTEN_JAR" ]; then
+ echo "ERROR: No Gluten bundle jar found under $GITHUB_WORKSPACE/gluten-bundle/" >&2
+ ls -la "$GITHUB_WORKSPACE/gluten-bundle/" || true
+ exit 1
+ fi
+ echo "Using Gluten bundle: $GLUTEN_JAR"
+ bash "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/setup-delta.sh" \
+ "${{ steps.resolve.outputs.delta_ref }}" \
+ "$GITHUB_WORKSPACE/delta" \
+ "$GLUTEN_JAR" \
+ "$GITHUB_WORKSPACE"
+
+ - name: Run Delta spark module tests (shard ${{ matrix.shard }} / ${{ env.DELTA_NUM_SHARDS }})
+ env:
+ NUM_SHARDS: ${{ env.DELTA_NUM_SHARDS }}
+ TEST_PARALLELISM_COUNT: ${{ steps.resolve.outputs.test_parallelism }}
+ # Required by Delta to enable testing-only code paths
+ # (see delta build.sbt: "Test / envVars += DELTA_TESTING -> 1").
+ DELTA_TESTING: '1'
+ # JDK 17 + Gluten/Arrow/Netty requires extra --add-opens and the
+ # `io.netty.tryReflectionSetAccessible` system property; otherwise
+ # the forked test JVM fails with
+ # java.lang.UnsupportedOperationException: sun.misc.Unsafe or
+ # java.nio.DirectByteBuffer.<init>(long, int) not available
+ # as soon as Gluten's bundled Arrow allocator initializes Netty
+ # direct buffers. Delta's own `Test / javaOptions` (see
+ # project/CrossSparkVersions.scala `java17TestSettings`) sets the
+ # base add-opens but NOT the Netty property -- Delta's own tests
+ # don't load Arrow/Netty buffers in a way that triggers it.
+ #
+ # Use JAVA_TOOL_OPTIONS so the flags propagate to BOTH the sbt
+ # launcher JVM and the forked test JVM (sbt forks tests and the
+ # child inherits the parent's env). The set below mirrors
+ # `extraJavaTestArgs` from Gluten's own root pom.xml (the
+ # canonical Gluten test JVM flag set).
+ #
+ # NOTE: we deliberately do NOT put `-Xmx` here. JAVA_TOOL_OPTIONS
+ # is processed BEFORE the JVM command line, so Delta's explicit
+ # `-Xmx1024m` (set in build.sbt `Test / javaOptions`) would still
+ # win (last `-Xmx` wins). The forked-test-JVM heap is bumped via
+ # an sbt `set spark / Test / javaOptions ++= ...` command below,
+ # which APPENDS to Delta's own seq -- so our `-Xmx` lands AFTER
+ # `-Xmx1024m` and wins.
+ JAVA_TOOL_OPTIONS: >-
+ -XX:+IgnoreUnrecognizedVMOptions
+ --add-opens=java.base/java.lang=ALL-UNNAMED
+ --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
+ --add-opens=java.base/java.lang.reflect=ALL-UNNAMED
+ --add-opens=java.base/java.io=ALL-UNNAMED
+ --add-opens=java.base/java.net=ALL-UNNAMED
+ --add-opens=java.base/java.nio=ALL-UNNAMED
+ --add-opens=java.base/java.util=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
+ --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.cs=ALL-UNNAMED
+ --add-opens=java.base/sun.security.action=ALL-UNNAMED
+ --add-opens=java.base/sun.util.calendar=ALL-UNNAMED
+ -Djdk.reflect.useDirectMethodHandle=false
+ -Dio.netty.tryReflectionSetAccessible=true
+ -Dfile.encoding=UTF-8
+ run: |
+ set -euo pipefail
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ cd "$GITHUB_WORKSPACE/delta"
+ chmod +x build/sbt
+ # Only run the unified `spark` sbt project, NOT `sparkGroup/test` --
+ # `sparkGroup` aggregates many other projects (sparkV2, contribs,
+ # sharing, connect*, ...) that are out of scope for this pipeline.
+ #
+ # JVM heap layout -- two memory consumers on the ~16G runner:
+ # * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to
+ # return idle memory during the (long) test phase via G1 periodic GC
+ # (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the
+ # busy fork doesn't suppress it; full STW via
+ # -G1PeriodicGCInvokesConcurrent) that uncommits to a tight free ratio
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3561413163
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,208 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# init + shallow fetch resolves a tag, branch OR commit SHA in a single path
+# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the
+# destructive `rm -rf "$DELTA_DIR"` it required. `--` terminates options so a
+# DELTA_REF starting with `-` can't be misread as a git flag (this script is
+# workflow_dispatch-runnable with a user-supplied ref).
+git init -q "$DELTA_DIR"
+git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git
+git -C "$DELTA_DIR" fetch -q --depth 1 origin -- "$DELTA_REF"
+git -C "$DELTA_DIR" checkout -q FETCH_HEAD
+git -C "$DELTA_DIR" --no-pager log -1 --oneline
+echo "::endgroup::"
+
+echo "::group::Injecting Gluten bundle jar onto the spark project's TEST classpath"
+# The Gluten bundle jar must be on the spark project's TEST runtime classpath
+# (so DeltaSQLCommandTest can load org.apache.gluten.GlutenPlugin by name) but
+# NOT on the COMPILE classpath of `sparkV1`, which is the project that holds
+# Delta's main sources. The bundle's transitive contents include extra symbols
+# under `org.apache.spark.sql` that collide with Delta's main sources -- e.g.
+# MergeOutputGeneration.scala imports both `org.apache.spark.sql._` and
+# `org.apache.spark.sql.delta.ClassicColumnConversions._`, and would then fail
+# with `reference to expression is ambiguous`.
+#
+# sbt auto-scans `<baseDirectory>/lib` via `unmanagedBase`. Two relevant
+# projects in Delta v4.2.0 have a `lib/` baseDirectory:
+# - sparkV1: `project in file("spark")` -> spark/lib
+# - spark : `project in file("spark-unified")` -> spark-unified/lib
+# unmanagedJars are project-scoped (NOT inherited by dependents), so dropping
+# the bundle into spark-unified/lib/ adds it to the unified `spark` project's
+# Compile *and* Test classpaths -- but NOT to sparkV1's. That's exactly what
+# we want:
+# * sparkV1/Compile sees ONLY Delta's regular deps -> Delta main compiles.
+# * spark/Test/fullClasspath sees the bundle -> tests load GlutenPlugin.
+# (Verified empirically: with bundle only in spark-unified/lib/, sbt's
+# `show sparkV1/Compile/dependencyClasspath` excludes the bundle and
+# `show spark/Test/fullClasspath` includes it.)
+#
+# We deliberately do NOT also drop the bundle into spark/lib/, which is what
+# caused the previous compile failure: spark/lib/ is sparkV1's unmanagedBase,
+# and putting the bundle there would re-introduce the ambiguity errors.
+SPARK_UNIFIED_LIB="$DELTA_DIR/spark-unified/lib"
+mkdir -p "$SPARK_UNIFIED_LIB"
+cp "$GLUTEN_BUNDLE_JAR" "$SPARK_UNIFIED_LIB/gluten-velox-bundle.jar"
+ls -lh "$SPARK_UNIFIED_LIB"
+echo "::endgroup::"
+
+echo "::group::Patching DeltaSQLCommandTest to enable Gluten plugin"
+TARGET="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$TARGET" ]; then
+ echo "Expected file not found in Delta clone: $TARGET" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
Review Comment:
This is an error path, but the second message is written to stdout instead of stderr. For consistency (and so logs/CI annotations capture it reliably), redirect it to stderr as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3555899244
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,747 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline file exists but is empty (not yet bootstrapped) the mode
+ automatically degrades to ``seed`` so the first run is never spuriously red.
+ A *missing* ``--known-failures`` file is treated as a configuration error
+ (the gate fails) so a mis-referenced path can't silently pass.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard). Pass ``--expected-shards N``
+ to fail when fewer than ``N`` shards contributed gate lists (a shard that
+ died before writing them), so an incomplete baseline is never produced.
+
+Flaky quarantine (``--flaky-tests``)
+ Some Delta-on-Gluten failures are non-deterministic (e.g. a native bug that
+ only triggers on certain runtime plans), so they are neither a stable pass
+ nor a stable failure and cannot live in the baseline: baselining them turns
+ the gate red on every run where they pass, and leaving them out turns it red
+ on every run where they fail. ``flaky-tests.txt`` quarantines them -- a
+ quarantined test never counts as a regression (when it fails) nor as
+ now-passing (when it passes), and is excluded from the regenerated baseline.
+ Its SUITE is an fnmatch glob (so one line covers a root-cause family across
+ generated suite variants); its TEST name is matched exactly.
+
+Flaky quarantine by error signature (``--flaky-error-patterns``)
+ When a failure is caused by a known nondeterministic bug that surfaces on a
+ *different test each run* (e.g. the native Delta DV bitmap row-index error),
+ matching by test name is whack-a-mole. ``flaky-error-patterns.txt`` instead
+ quarantines by root cause: each line is a regex matched against a failed
+ test's <failure>/<error> text, and any failure that matches is treated as
+ flaky regardless of which test it landed on.
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import fnmatch
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+
+class NoReportsError(RuntimeError):
+ """Raised when no JUnit <testsuite> elements are found under reports_dir."""
+
+
+class CorruptReportError(NoReportsError):
+ """Raised when an expected JUnit report file (TEST-*.xml) fails to parse.
+
+ Subclasses NoReportsError so the enforce/seed handler treats a truncated
+ report as a hard data error (exit 2) instead of silently dropping the
+ suite's results and letting the gate pass on partial data.
+ """
+
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def make_is_flaky(flaky_entries):
+ """Build a predicate that matches a (suite, test) tuple against flaky entries.
+
+ A flaky entry quarantines a test whose failure is known to be non-deterministic
+ (see flaky-tests.txt). The entry's SUITE is treated as an fnmatch glob so a
+ single line can cover a root-cause family across generated suite variants
+ (e.g. ``*DVs*Suite`` matches every deletion-vector merge suite, ``*`` matches
+ any suite); the TEST name is matched exactly (test names are freeform and may
+ contain glob metacharacters, so they are never globbed).
+ """
+ exact = set()
+ globbed = []
+ for suite, test in flaky_entries:
+ if any(ch in suite for ch in "*?["):
+ globbed.append((suite, test))
+ else:
+ exact.add((suite, test))
+
+ def is_flaky(entry):
+ if entry in exact:
+ return True
+ suite, test = entry
+ for glob_suite, glob_test in globbed:
+ if test == glob_test and fnmatch.fnmatchcase(suite, glob_suite):
+ return True
+ return False
+
+ return is_flaky
+
+
+def load_patterns(path):
+ """Load flaky-error regex patterns from a file (one per line).
+
+ Blank lines and ``#`` comments are ignored. Each remaining line is compiled
+ as a case-sensitive regex. These match the FAILURE TEXT of a failed test
+ (its JUnit <failure>/<error> message + stack), so a test that fails with a
+ known-nondeterministic native error (e.g. the Delta DV bitmap row-index bug)
+ can be quarantined by root cause instead of by exact test name.
+ """
+ patterns = []
+ if not path or not os.path.exists(path):
+ return patterns
+ with open(path, encoding="utf-8") as fh:
+ for line in fh:
+ line = line.rstrip("\n")
+ if not line.strip() or line.lstrip().startswith("#"):
+ continue
+ patterns.append(re.compile(line))
+ return patterns
Review Comment:
Don't need
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5131363212
@philo-he @felipepessoto
> Could we use Apache Stash to cache the native lib so it can be shared across workflows?
+1. We could upload the native lib artifact and share download it in the new delta pipeline. Should also add a fallback native build in case there is a cache miss.
Is there a easy way to cover multiple Spark versions for these delta tests? The delta community has moved to spark-4.x but for Gluten many are still on Spark-3.5.
The other issue is the 4 shard based job is taking 2 hours for each, it's better to split into 8 shard so it can be a faster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5074988161
@philo-he I believe I have covered all the concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3502473808
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,484 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline file exists but is empty (not yet bootstrapped) the mode
+ automatically degrades to ``seed`` so the first run is never spuriously red.
+ A *missing* ``--known-failures`` file is treated as a configuration error
+ (the gate fails) so a mis-referenced path can't silently pass.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard).
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import glob
+import os
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped) sets of (suite, test) tuples. A test is
+ 'failed' if its <testcase> has a <failure> or <error> child, 'skipped' if
+ it has a <skipped> child, otherwise 'passed'. Suite-level aborts (a
+ <testsuite> reporting errors/failures with no failing <testcase>) are
+ recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = (suite, name)
+ tags = _child_local_tags(tc)
+ if "failure" in tags or "error" in tags:
+ failed.add(key)
+ suite_has_failing_tc = True
+ elif "skipped" in tags:
+ skipped.add(key)
+ else:
+ passed.add(key)
+
+ # Suite-level abort: counters say something failed but no testcase
+ # carried the failure (the suite blew up in beforeAll/constructor).
+ # Record a
+ # synthetic entry so the regression is visible.
+ try:
+ errors = int(ts.get("errors", "0") or "0")
+ failures = int(ts.get("failures", "0") or "0")
+ except ValueError:
+ errors = failures = 0
+ if (errors + failures) > 0 and not suite_has_failing_tc:
+ failed.add((suite_name, SUITE_ABORTED))
+
+ if not parsed_any:
+ eprint(
+ "WARNING: no JUnit <testsuite> elements found under {}".format(reports_dir)
+ )
+
Review Comment:
Done in 6fe545120. parse_reports now raises NoReportsError and the enforce/seed run returns exit code 2 when no <testsuite> elements are parsed, instead of warning and returning empty sets. A misconfigured reports dir (or a reporter that produced nothing) now fails the shard loudly rather than passing the gate on an empty result set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3554133732
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,446 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib artifact it already built, so the expensive
+ # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a
+ # called workflow runs as part of the caller run), so the jobs below download it
+ # by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
Review Comment:
Not doing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5145623546
@philo-he @zhouyuan This is green now and I think it's ready — the pipeline is fully self-contained and the last run passed end to end (8/8 shards, slowest 102 min, ~10 job-hours).
What's in this PR:
- **Own workflow.** Delta CI no longer runs inside `velox_backend_x86.yml`, so a change to the Delta pipeline no longer drags in the ~50-job TPC-H/DS + Spark-UT matrix (~1750 runner-min). Per-PR it's gated by a `paths:` filter, plus a nightly run at 05:00 UTC for everything the filter skips.
- **8 shards x 4 forks**, matching delta-io/delta's own `spark_test.yaml`. Measured: slowest shard 163 → 102 min (-37%) for +5% billed hours.
- Baseline at 735 known failures, 0 regressions.
Proposed follow-ups, happy to file issues for these:
1. **Share the native lib** between the two workflows (~10 min on `gluten-delta` PRs, which currently build it twice). Needs a cache key covering `cpp/**`, build flags and the base image — a stale hit there means silently testing the wrong binary.
2. **Spark 3.5 leg**, nightly-only at first. The blocker is the baseline: each Spark/Delta combination needs its own file and a bootstrap run. Once 3.5 is covered here, the imported Delta suite copies @malinjawi identified become removable.
3. **`run-delta-ci` label**, if you want the opt-in back. I dropped it since it can't be expressed as a `paths:` filter and the label didn't exist on the repo; `workflow_dispatch` covers the same need today.
Does this look right to you both?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3660529251
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,208 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# init + shallow fetch resolves a tag, branch OR commit SHA in a single path
+# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the
+# destructive `rm -rf "$DELTA_DIR"` it required. `--` terminates options so a
+# DELTA_REF starting with `-` can't be misread as a git flag (this script is
+# workflow_dispatch-runnable with a user-supplied ref).
+git init -q "$DELTA_DIR"
+git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git
+git -C "$DELTA_DIR" fetch -q --depth 1 origin -- "$DELTA_REF"
+git -C "$DELTA_DIR" checkout -q FETCH_HEAD
+git -C "$DELTA_DIR" --no-pager log -1 --oneline
+echo "::endgroup::"
+
+echo "::group::Injecting Gluten bundle jar onto the spark project's TEST classpath"
+# The Gluten bundle jar must be on the spark project's TEST runtime classpath
+# (so DeltaSQLCommandTest can load org.apache.gluten.GlutenPlugin by name) but
+# NOT on the COMPILE classpath of `sparkV1`, which is the project that holds
+# Delta's main sources. The bundle's transitive contents include extra symbols
+# under `org.apache.spark.sql` that collide with Delta's main sources -- e.g.
+# MergeOutputGeneration.scala imports both `org.apache.spark.sql._` and
+# `org.apache.spark.sql.delta.ClassicColumnConversions._`, and would then fail
+# with `reference to expression is ambiguous`.
+#
+# sbt auto-scans `<baseDirectory>/lib` via `unmanagedBase`. Two relevant
+# projects in Delta v4.2.0 have a `lib/` baseDirectory:
+# - sparkV1: `project in file("spark")` -> spark/lib
+# - spark : `project in file("spark-unified")` -> spark-unified/lib
+# unmanagedJars are project-scoped (NOT inherited by dependents), so dropping
+# the bundle into spark-unified/lib/ adds it to the unified `spark` project's
+# Compile *and* Test classpaths -- but NOT to sparkV1's. That's exactly what
+# we want:
+# * sparkV1/Compile sees ONLY Delta's regular deps -> Delta main compiles.
+# * spark/Test/fullClasspath sees the bundle -> tests load GlutenPlugin.
+# (Verified empirically: with bundle only in spark-unified/lib/, sbt's
+# `show sparkV1/Compile/dependencyClasspath` excludes the bundle and
+# `show spark/Test/fullClasspath` includes it.)
+#
+# We deliberately do NOT also drop the bundle into spark/lib/, which is what
+# caused the previous compile failure: spark/lib/ is sparkV1's unmanagedBase,
+# and putting the bundle there would re-introduce the ambiguity errors.
+SPARK_UNIFIED_LIB="$DELTA_DIR/spark-unified/lib"
+mkdir -p "$SPARK_UNIFIED_LIB"
+cp "$GLUTEN_BUNDLE_JAR" "$SPARK_UNIFIED_LIB/gluten-velox-bundle.jar"
+ls -lh "$SPARK_UNIFIED_LIB"
+echo "::endgroup::"
+
+echo "::group::Patching DeltaSQLCommandTest to enable Gluten plugin"
+TARGET="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$TARGET" ]; then
+ echo "Expected file not found in Delta clone: $TARGET" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
+ exit 1
+fi
+cp "$PATCH_SOURCE" "$TARGET"
+echo "Patched $TARGET"
+echo "--- diff vs. upstream ---"
+git -C "$DELTA_DIR" --no-pager diff -- "spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala" || true
+echo "::endgroup::"
+
+# Delta's tests collect file-source scans by matching the concrete
+# `FileSourceScanExec` case class; Gluten offloads the scan to
+# DeltaScanTransformer, a `FileSourceScanLike` sibling, so those matches miss
+# (`scala.MatchError: List()`, empty partition filters, broken column-pruning /
+# scan-metric checks across many suites). delta-io/delta#7104 and #7105 widen the
+# matches to the shared `FileSourceScanLike` interface that both the vanilla and
+# Gluten scans implement (behavior-preserving for vanilla). Both are merged
+# upstream but land after the pinned DELTA_REF (v4.2.0), so apply them here; once
+# DELTA_REF includes a commit its cherry-pick is a clean no-op and the call can go.
+#
+# Depth-2 fetch brings each fix commit and its parent, which cherry-pick needs to
+# diff against (a depth-1 fetch grafts the parent away); `-n` stages the change
+# without requiring a committer identity.
+cherry_pick_delta_fix() {
+ local sha="$1" pr="$2"
+ echo "Cherry-picking delta-io/delta${pr}"
+ git -C "$DELTA_DIR" fetch --quiet --depth 2 origin "$sha"
+ git -C "$DELTA_DIR" cherry-pick -n "$sha"
+}
+
+echo "::group::Cherry-picking upstream Delta FileSourceScanLike test fixes"
+cherry_pick_delta_fix 46bd45d57eadd7e528002a0ae7bd36ce5a456eca "#7104 (ScanReportHelper.collectScans)"
+cherry_pick_delta_fix 959e00e15f41f56afc1c9bb95d160c55c6dc7068 "#7105 (9 more test suites)"
Review Comment:
Good catch — this is a real latent bug. `setup-delta.sh` runs under `set -euo pipefail`, and cherry-picking a fix that the pinned `DELTA_REF` already contains exits non-zero (empty/conflicting patch), which aborts the whole setup. So bumping `DELTA_REF` past #7104/#7105 would break the pipeline — the old "clean no-op" comment was wrong.
Fixed in 4bedc87c4: `cherry_pick_delta_fix` now attempts the cherry-pick and, on failure, recovers only the paths that fix touches (`git diff-tree` → per-file `reset`+`checkout`, then `cherry-pick --quit`) and continues, leaving the DeltaSQLCommandTest patch and the bundle jar intact. It's self-correcting: if a fix is genuinely still needed, the `FileSourceScanLike` failures it prevents resurface as gate regressions rather than being hidden by a hard abort.
I went with recover-on-failure rather than `merge-base --is-ancestor` because the Delta clone is shallow (depth 1), so ancestry can't see past the graft and `is-ancestor` returns the wrong answer here. (A reverse-apply-check is also fragile when the newer ref carries the fix plus adjacent edits.) Verified against a clean base (fix applies), an already-containing ref with nearby changes (skips cleanly, no conflict state left behind), and a divergent ref; `shellcheck` + `bash -n` clean.
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -101,6 +106,70 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR
+ # touches high-signal Delta paths -- the Delta integration code
+ # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own
+ # files -- or carries the `run-delta-ci` opt-in label. Changes to general
+ # Velox/core/native code can also affect Delta offload but are touched
+ # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml
+ # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage
+ # down. NOTE: the label is read from the event that triggered this run, so add
+ # it before/with a push; labeling an already-finished run needs a new push.
+ delta-changes:
+ runs-on: ubuntu-22.04
+ outputs:
+ run_delta: ${{ steps.filter.outputs.run_delta }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ - name: Detect Delta-relevant changes / opt-in label
+ id: filter
+ env:
+ HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
Review Comment:
This workflow is triggered `on: pull_request` only, so `github.event.pull_request.*` is always present. And in GitHub Actions, reading a property of a null context yields an empty string rather than failing evaluation, so there's no eval-time error to guard against — the shell fail-open branch is just belt-and-suspenders. An `event_name == 'pull_request'` guard would always be true for this workflow, so I've left it as-is.
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,459 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib artifact it already built, so the expensive
+ # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a
+ # called workflow runs as part of the caller run), so the jobs below download it
+ # by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+ # Nightly full run against the latest default branch. The per-PR entry point
+ # (velox_backend_x86.yml) now runs the Delta suite only when a PR touches
+ # Delta-relevant paths (or carries the opt-in label), to save GHA minutes; this
+ # scheduled run keeps full coverage once a day so rarer regressions are still
+ # caught. It builds its own native lib (build-native-lib-centos-7 below) since
+ # there is no caller to provide one, and uses the workflow's default inputs.
+ schedule:
+ - cron: '0 5 * * *'
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
Review Comment:
This env var isn't introduced by this change — it predates it and is set in ~10 workflows across the repo, because several pinned actions still run on Node 20. Dropping it only here would be inconsistent and would surface hard failures / deprecation noise for those actions. Migrating actions off Node 20 is a repo-wide change better handled separately.
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,459 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib artifact it already built, so the expensive
+ # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a
+ # called workflow runs as part of the caller run), so the jobs below download it
+ # by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+ # Nightly full run against the latest default branch. The per-PR entry point
+ # (velox_backend_x86.yml) now runs the Delta suite only when a PR touches
+ # Delta-relevant paths (or carries the opt-in label), to save GHA minutes; this
+ # scheduled run keeps full coverage once a day so rarer regressions are still
+ # caught. It builds its own native lib (build-native-lib-centos-7 below) since
+ # there is no caller to provide one, and uses the workflow's default inputs.
+ schedule:
+ - cron: '0 5 * * *'
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. `spark_version` is the single source of truth for the
+ # Spark version: it drives the Gluten bundle profile (-Pspark-<v>), the bundle
+ # jar name, and Delta's -DsparkVersion, so the tests always run against a bundle
+ # built for the same Spark version (no separate value to keep in sync). Scala
+ # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from
+ # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a
+ # compatible delta_ref.
+ GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version || '4.1' }}
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version || '4.1' }}
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone runs (workflow_dispatch + nightly schedule) build the native lib
+ # here. When called by velox_backend_x86.yml the caller already built it and
+ # passes it as an input, so this job is skipped and the duplicate build avoided.
+ if: github.event_name == 'workflow_dispatch' || github.event_name == 'schedule'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
Review Comment:
The `apache/gluten:*` images are referenced by tag throughout every Gluten workflow (dozens of uses). Pinning by digest is a reasonable supply-chain improvement, but doing it only in the Delta workflow would be inconsistent — better handled as a repo-wide policy change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3660732584
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,230 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# init + shallow fetch resolves a tag, branch OR commit SHA in a single path
+# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the
+# destructive `rm -rf "$DELTA_DIR"` it required. `--` terminates options so a
+# DELTA_REF starting with `-` can't be misread as a git flag (this script is
+# workflow_dispatch-runnable with a user-supplied ref).
+git init -q "$DELTA_DIR"
+git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git
+git -C "$DELTA_DIR" fetch -q --depth 1 origin -- "$DELTA_REF"
+git -C "$DELTA_DIR" checkout -q FETCH_HEAD
Review Comment:
Valid — confirmed it hard-fails, so fixed in 85fdc62ed.
`git init` is happily idempotent, but `git remote add origin` exits 3 with `error: remote origin already exists.` on a second run, which under `set -euo pipefail` aborts the script. Reproduced locally: first run OK, second run exit 3.
The clone block now drops the remote before re-adding it and force-checks-out `FETCH_HEAD`, so a re-run (local, or a CI re-run on a runner that kept the workspace) resumes instead of dying, and a partial/dirty previous tree is reset rather than blocking the checkout. Verified across three consecutive runs including one with a modified tracked file and a stray untracked file.
I deliberately did not go back to deleting `DELTA_DIR` — an earlier revision of this script had an `rm -rf "$DELTA_DIR"` and it was removed on purpose. Making the steps idempotent gets the same robustness without a destructive delete, and it's safe here because the bundle jar and the source patches are applied *after* this block, so nothing worth keeping exists yet.
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,769 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline file exists but is empty (not yet bootstrapped) the mode
+ automatically degrades to ``seed`` so the first run is never spuriously red.
+ A *missing* ``--known-failures`` file is treated as a configuration error
+ (the gate fails) so a mis-referenced path can't silently pass.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard). Pass ``--expected-shards N``
+ to fail when fewer than ``N`` shards contributed gate lists (a shard that
+ died before writing them), so an incomplete baseline is never produced.
+
+Flaky quarantine (``--flaky-tests``)
+ Some Delta-on-Gluten failures are non-deterministic (e.g. a native bug that
+ only triggers on certain runtime plans), so they are neither a stable pass
+ nor a stable failure and cannot live in the baseline: baselining them turns
+ the gate red on every run where they pass, and leaving them out turns it red
+ on every run where they fail. ``flaky-tests.txt`` quarantines them -- a
+ quarantined test never counts as a regression (when it fails) nor as
+ now-passing (when it passes), and is excluded from the regenerated baseline.
+ Its SUITE is an fnmatch glob (so one line covers a root-cause family across
+ generated suite variants); its TEST name is matched exactly.
+
+Flaky quarantine by error signature (``--flaky-error-patterns``)
+ When a failure is caused by a known nondeterministic bug that surfaces on a
+ *different test each run* (e.g. the native Delta DV bitmap row-index error),
+ matching by test name is whack-a-mole. ``flaky-error-patterns.txt`` instead
+ quarantines by root cause: each line is a regex matched against a failed
+ test's <failure>/<error> text, and any failure that matches is treated as
+ flaky regardless of which test it landed on.
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import fnmatch
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+
+class NoReportsError(RuntimeError):
+ """Raised when no JUnit <testsuite> elements are found under reports_dir."""
+
+
+class CorruptReportError(NoReportsError):
+ """Raised when an expected JUnit report file (TEST-*.xml) fails to parse.
+
+ Subclasses NoReportsError so the enforce/seed handler treats a truncated
+ report as a hard data error (exit 2) instead of silently dropping the
+ suite's results and letting the gate pass on partial data.
+ """
+
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def normalize_key(suite, test):
+ """Normalize a (suite, test) key parsed from JUnit XML to match baseline keys.
+
+ Baseline/flaky entries round-trip through write_entries (which collapses CR/LF
+ in the test name to spaces) and parse_entry (which strips the whole
+ ``suite#test`` line). A raw XML name carrying a trailing newline or
+ surrounding whitespace would therefore never equal its normalized baseline
+ entry: the gate would keep reporting it as a REGRESSION, and the
+ copy-pasteable line it prints could never suppress it (load strips it back).
+ Delta test names are freeform, so a version bump could introduce exactly that.
+ Applying the identical format+parse round-trip here keeps the two sides in
+ sync (and is a no-op for the normal, whitespace-free names).
+ """
+ safe_test = (test or "").replace("\r", " ").replace("\n", " ")
+ normalized = parse_entry(format_entry(suite or "", safe_test))
+ # parse_entry only returns None for a blank/comment line, which a real
+ # testcase key is not; fall back to a bare strip to keep this total.
+ if normalized is None:
+ return ((suite or "").strip(), safe_test.strip())
+ return normalized
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def make_is_flaky(flaky_entries):
+ """Build a predicate that matches a (suite, test) tuple against flaky entries.
+
+ A flaky entry quarantines a test whose failure is known to be non-deterministic
+ (see flaky-tests.txt). The entry's SUITE is treated as an fnmatch glob so a
+ single line can cover a root-cause family across generated suite variants
+ (e.g. ``*DVs*Suite`` matches every deletion-vector merge suite, ``*`` matches
+ any suite); the TEST name is matched exactly (test names are freeform and may
+ contain glob metacharacters, so they are never globbed).
+ """
+ exact = set()
+ globbed = []
+ for suite, test in flaky_entries:
+ if any(ch in suite for ch in "*?["):
+ globbed.append((suite, test))
+ else:
+ exact.add((suite, test))
+
+ def is_flaky(entry):
+ if entry in exact:
+ return True
+ suite, test = entry
+ for glob_suite, glob_test in globbed:
+ if test == glob_test and fnmatch.fnmatchcase(suite, glob_suite):
+ return True
+ return False
+
+ return is_flaky
+
+
+def load_patterns(path):
+ """Load flaky-error regex patterns from a file (one per line).
+
+ Blank lines and ``#`` comments are ignored. Each remaining line is compiled
+ as a case-sensitive regex. These match the FAILURE TEXT of a failed test
+ (its JUnit <failure>/<error> message + stack), so a test that fails with a
+ known-nondeterministic native error (e.g. the Delta DV bitmap row-index bug)
+ can be quarantined by root cause instead of by exact test name.
+ """
+ patterns = []
+ if not path or not os.path.exists(path):
+ return patterns
+ with open(path, encoding="utf-8") as fh:
+ for line in fh:
+ line = line.rstrip("\n")
+ if not line.strip() or line.lstrip().startswith("#"):
+ continue
+ patterns.append(re.compile(line))
+ return patterns
+
+
+def make_signature_matcher(patterns):
+ """Return a predicate matching a failure text against any flaky-error pattern."""
+
+ def matches(text):
+ if not text:
+ return False
+ return any(p.search(text) for p in patterns)
+
+ return matches
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def _failure_text(tc):
+ """Concatenate the message attribute + body text of a testcase's
+ <failure>/<error> children, for error-signature matching."""
+ parts = []
+ for c in tc:
+ if c.tag.split("}")[-1] in ("failure", "error"):
+ msg = c.get("message")
+ if msg:
+ parts.append(msg)
+ if c.text:
+ parts.append(c.text)
+ return "\n".join(parts)
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped, fail_texts). The first three are sets of
+ (suite, test) tuples; fail_texts maps each failed (suite, test) to its
+ combined <failure>/<error> message + stack text (used for error-signature
+ quarantine). A test is 'failed' if its <testcase> has a <failure> or <error>
+ child, 'skipped' if it has a <skipped> child, otherwise 'passed'. Suite-level
+ aborts (a <testsuite> reporting errors/failures with no failing <testcase>)
+ are recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+ fail_texts = {}
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ # A TEST-*.xml that fails to parse is almost always a report truncated
+ # when a forked test JVM was killed mid-write (e.g. OOM). Silently
+ # skipping it drops that suite's results and could let the gate go
+ # green on partial data, so fail hard for report files. Other XML that
+ # merely matched the broad `target/**` glob is still skipped.
+ if os.path.basename(xml_file).startswith("TEST-"):
+ raise CorruptReportError(
+ "corrupt or truncated JUnit report {}: {}. Refusing to "
+ "evaluate the gate on partial data.".format(xml_file, exc)
+ )
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = normalize_key(suite, name)
+ tags = _child_local_tags(tc)
+ if "failure" in tags or "error" in tags:
+ failed.add(key)
+ suite_has_failing_tc = True
+ fail_texts[key] = _failure_text(tc)
+ elif "skipped" in tags:
+ skipped.add(key)
+ else:
+ passed.add(key)
+
+ # Suite-level abort: counters say something failed but no testcase
+ # carried the failure (the suite blew up in beforeAll/constructor).
+ # Record a
+ # synthetic entry so the regression is visible.
+ try:
+ errors = int(ts.get("errors", "0") or "0")
+ failures = int(ts.get("failures", "0") or "0")
+ except ValueError:
+ errors = failures = 0
+ if (errors + failures) > 0 and not suite_has_failing_tc:
+ failed.add(normalize_key(suite_name, SUITE_ABORTED))
+
+ if not parsed_any:
+ raise NoReportsError(
+ "No JUnit <testsuite> elements found under {}. The test reports are "
+ "missing or in an unexpected format -- refusing to evaluate the gate "
+ "on an empty result set (this would otherwise pass silently).".format(
+ reports_dir
+ )
+ )
+
+ # A test can't be both passed and failed; failure wins. Skipped only counts
+ # if the test was not otherwise seen (e.g. retried).
+ passed -= failed
+ skipped -= failed
+ skipped -= passed
+ return passed, failed, skipped, fail_texts
+
+
+# --------------------------------------------------------------------------- #
+# Reporting helpers
+# --------------------------------------------------------------------------- #
+def _summary_sink():
+ """Return a writer that mirrors to GITHUB_STEP_SUMMARY when available."""
+ path = os.environ.get("GITHUB_STEP_SUMMARY")
+ handle = open(path, "a", encoding="utf-8") if path else None
+
+ def write(line=""):
+ print(line)
+ if handle:
+ handle.write(line + "\n")
+
+ return write, handle
+
+
+def _print_block(write, title, entries, limit=50):
+ write("")
+ write("### {} ({})".format(title, len(entries)))
+ if not entries:
+ return
+ write("")
+ write("```")
+ for i, (suite, test) in enumerate(sorted(entries)):
+ if i >= limit:
+ write("... and {} more".format(len(entries) - limit))
+ break
+ write(format_entry(suite, test))
+ write("```")
+
+
+# --------------------------------------------------------------------------- #
+# Modes
+# --------------------------------------------------------------------------- #
+def run_enforce(args):
+ # In enforce mode a missing baseline file would make load_entries() return an
+ # empty set, silently degrading to seed mode and passing the gate without
+ # enforcing anything. Treat a missing path as a configuration error; an
+ # existing-but-empty file is still allowed (it legitimately seeds).
+ if args.mode == "enforce" and (
+ not args.known_failures or not os.path.exists(args.known_failures)
+ ):
+ eprint(
+ "ERROR: --known-failures '{}' does not exist. In enforce mode the "
+ "baseline file must exist (an existing-but-empty file is allowed and "
+ "triggers seed mode). Refusing to silently pass.".format(
+ args.known_failures
+ )
+ )
+ return 2
+ baseline = load_entries(args.known_failures)
+ name_flaky = make_is_flaky(load_entries(args.flaky_tests))
+ sig_matches = make_signature_matcher(load_patterns(args.flaky_error_patterns))
+ try:
+ passed, failed, skipped, fail_texts = parse_reports(args.reports_dir)
+ except NoReportsError as exc:
+ eprint("ERROR: {}".format(exc))
+ return 2
+
+ # A test is quarantined if its NAME is in flaky-tests.txt, or its failure
+ # TEXT matches a flaky-error signature (e.g. the native DV bitmap row-index
+ # bug that hits a different DV-merge test each run).
+ def sig_flaky(e):
+ return e in fail_texts and sig_matches(fail_texts[e])
+
+ def flaky_is(e):
+ return name_flaky(e) or sig_flaky(e)
+
+ # Always emit this shard's artifacts for the aggregation job. Signature-flaky
+ # failures are dropped from failures-out: the aggregate job works off these
+ # text-less lists and cannot re-derive the signature match, so excluding them
+ # here keeps the regenerated baseline from absorbing a flaky failure. Name-
+ # flaky entries stay (the aggregate re-filters them via flaky-tests.txt).
+ if args.failures_out:
+ write_entries(args.failures_out, {e for e in failed if not sig_flaky(e)})
+ if args.ran_out:
+ write_entries(args.ran_out, passed | failed)
Review Comment:
The underlying observation is right and is now fixed in 85fdc62ed, though I implemented it differently than suggested, because adding `skipped` to the `ran-out` set would introduce a worse bug.
`fixed` (now-passing) is derived from `baseline & (union_ran - union_failed)`. If skipped tests were part of `union_ran`, a baseline test that merely got skipped would be reported as **now-passing** — and since the nightly run uses `fail_on_fixed=true`, that would fail the build and demand removing the entry from the baseline, only for it to come back as a regression the next time it actually executes. I verified this: folding skips into `ran-out` yields `Now-passing (global) | 1` for a purely skipped test.
So instead each shard now writes a separate `--skipped-out` list, and the aggregate job:
- computes `stale = baseline - union_ran - union_skipped`, so "stale" means truly absent (your point);
- reports skipped baseline entries under their own "Skipped this run — NOT stale, leave them in the baseline" block;
- carries them over into the regenerated `known-failures.txt` (previously regeneration ran purely off `union_failed`, so a skipped known-failure was silently dropped — the same bug showing up in the artifact);
- leaves `fixed` keyed off pass/fail only, so a skip is never mistaken for a fix.
Missing `skipped-*.txt` (older artifacts) degrades gracefully to the previous behaviour, and the shard-completeness guard still keys off `failures-*`/`ran-*` only. Verified end-to-end: skip stays out of `ran-out`, stale = 0, now-passing = 0, entry preserved in the regenerated baseline, and legacy artifacts without the new file still aggregate cleanly.
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,459 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib artifact it already built, so the expensive
+ # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a
+ # called workflow runs as part of the caller run), so the jobs below download it
+ # by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+ # Nightly full run against the latest default branch. The per-PR entry point
+ # (velox_backend_x86.yml) now runs the Delta suite only when a PR touches
+ # Delta-relevant paths (or carries the opt-in label), to save GHA minutes; this
+ # scheduled run keeps full coverage once a day so rarer regressions are still
+ # caught. It builds its own native lib (build-native-lib-centos-7 below) since
+ # there is no caller to provide one, and uses the workflow's default inputs.
+ schedule:
+ - cron: '0 5 * * *'
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
Review Comment:
Same as [my earlier reply on this line](https://github.com/apache/gluten/pull/12388#discussion_r3659868690): this env var isn't introduced by this change — it predates it and is set in ~10 workflows across the repo, because several pinned actions still run on Node 20. Dropping it only here would be inconsistent and would surface hard failures / deprecation noise for those actions. Migrating actions off Node 20 is a repo-wide change better handled separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3533071294
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,683 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Delta `-DsparkVersion` value (must match the Gluten -P profile below)'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. Spark 4.1 + Scala 2.13 + JDK 17 matches Delta v4.2.0's
+ # default Spark version (4.1.0) from project/CrossSparkVersions.scala.
+ GLUTEN_SPARK_PROFILE: 'spark-4.1'
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: '4.1'
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Every input has a default (workflow_call + workflow_dispatch), so they
+ # are always set; just surface them as step outputs for the steps below.
+ env:
+ DELTA_REF: ${{ inputs.delta_ref }}
+ SPARK_VERSION: ${{ inputs.spark_version }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline }}
+ FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
+
+ - name: Download Gluten bundle jar
+ uses: actions/download-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: gluten-bundle
+
+ - name: Install minimal tools
+ run: |
+ set -euo pipefail
+ # apache/gluten:centos-9-jdk17 already has java-17, git, tar, a POSIX
+ # shell, and curl-minimal (which provides the `curl` command sbt's
+ # launcher needs). Install the rest of what Delta's build/sbt and the
+ # tests may need. We deliberately do NOT install the full `curl`
+ # package -- it conflicts with the pre-installed curl-minimal.
+ yum install -y java-17-openjdk-devel which findutils gzip python3
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ git --version
+ curl --version | head -n 1
+
+ - name: Cache sbt / Ivy / Coursier
+ uses: actions/cache@v4
+ with:
+ path: |
+ /root/.sbt
+ /root/.ivy2
+ /root/.cache/coursier
+ # Intentionally NOT keyed by ${{ matrix.shard }} -- all shards share
+ # the same dependency tree, so a single shared cache (with parallel
+ # save races resolved by GH on a first-write-wins basis) gives the
+ # best storage / hit-rate tradeoff.
+ key: delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-${{ env.DELTA_SCALA_VERSION }}
+ restore-keys: |
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-
+
+ - name: Clone and patch Delta
+ run: |
+ set -euo pipefail
+ # `2>/dev/null ... || true` keeps a no-match `ls` from aborting the step
+ # under `set -o pipefail`, so the explicit check below emits a clear
+ # error instead of a generic "cannot access".
+ GLUTEN_JAR=$(ls "$GITHUB_WORKSPACE"/gluten-bundle/gluten-velox-bundle-spark*_*-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$GLUTEN_JAR" ] || [ ! -f "$GLUTEN_JAR" ]; then
+ echo "ERROR: No Gluten bundle jar found under $GITHUB_WORKSPACE/gluten-bundle/" >&2
+ ls -la "$GITHUB_WORKSPACE/gluten-bundle/" || true
+ exit 1
+ fi
+ echo "Using Gluten bundle: $GLUTEN_JAR"
+ bash "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/setup-delta.sh" \
+ "${{ steps.resolve.outputs.delta_ref }}" \
+ "$GITHUB_WORKSPACE/delta" \
+ "$GLUTEN_JAR" \
+ "$GITHUB_WORKSPACE"
+
+ - name: Run Delta spark module tests (shard ${{ matrix.shard }} / ${{ env.DELTA_NUM_SHARDS }})
+ env:
+ NUM_SHARDS: ${{ env.DELTA_NUM_SHARDS }}
+ TEST_PARALLELISM_COUNT: ${{ steps.resolve.outputs.test_parallelism }}
+ # Required by Delta to enable testing-only code paths
+ # (see delta build.sbt: "Test / envVars += DELTA_TESTING -> 1").
+ DELTA_TESTING: '1'
+ # JDK 17 + Gluten/Arrow/Netty requires extra --add-opens and the
+ # `io.netty.tryReflectionSetAccessible` system property; otherwise
+ # the forked test JVM fails with
+ # java.lang.UnsupportedOperationException: sun.misc.Unsafe or
+ # java.nio.DirectByteBuffer.<init>(long, int) not available
+ # as soon as Gluten's bundled Arrow allocator initializes Netty
+ # direct buffers. Delta's own `Test / javaOptions` (see
+ # project/CrossSparkVersions.scala `java17TestSettings`) sets the
+ # base add-opens but NOT the Netty property -- Delta's own tests
+ # don't load Arrow/Netty buffers in a way that triggers it.
+ #
+ # Use JAVA_TOOL_OPTIONS so the flags propagate to BOTH the sbt
+ # launcher JVM and the forked test JVM (sbt forks tests and the
+ # child inherits the parent's env). The set below mirrors
+ # `extraJavaTestArgs` from Gluten's own root pom.xml (the
+ # canonical Gluten test JVM flag set).
+ #
+ # NOTE: we deliberately do NOT put `-Xmx` here. JAVA_TOOL_OPTIONS
+ # is processed BEFORE the JVM command line, so Delta's explicit
+ # `-Xmx1024m` (set in build.sbt `Test / javaOptions`) would still
+ # win (last `-Xmx` wins). The forked-test-JVM heap is bumped via
+ # an sbt `set spark / Test / javaOptions ++= ...` command below,
+ # which APPENDS to Delta's own seq -- so our `-Xmx` lands AFTER
+ # `-Xmx1024m` and wins.
+ JAVA_TOOL_OPTIONS: >-
+ -XX:+IgnoreUnrecognizedVMOptions
+ --add-opens=java.base/java.lang=ALL-UNNAMED
+ --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
+ --add-opens=java.base/java.lang.reflect=ALL-UNNAMED
+ --add-opens=java.base/java.io=ALL-UNNAMED
+ --add-opens=java.base/java.net=ALL-UNNAMED
+ --add-opens=java.base/java.nio=ALL-UNNAMED
+ --add-opens=java.base/java.util=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
+ --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.cs=ALL-UNNAMED
+ --add-opens=java.base/sun.security.action=ALL-UNNAMED
+ --add-opens=java.base/sun.util.calendar=ALL-UNNAMED
+ -Djdk.reflect.useDirectMethodHandle=false
+ -Dio.netty.tryReflectionSetAccessible=true
+ -Dfile.encoding=UTF-8
+ run: |
+ set -euo pipefail
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ cd "$GITHUB_WORKSPACE/delta"
+ chmod +x build/sbt
+ # Only run the unified `spark` sbt project, NOT `sparkGroup/test` --
+ # `sparkGroup` aggregates many other projects (sparkV2, contribs,
+ # sharing, connect*, ...) that are out of scope for this pipeline.
+ #
+ # JVM heap layout -- two memory consumers on the ~16G runner:
+ # * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to
+ # return idle memory during the (long) test phase via G1 periodic GC
+ # (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the
+ # busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent
+ # forces each periodic GC to a full STW collection) that uncommits to a
+ # tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low
+ # -Xms512m floor.
+ # Without this the idle launcher holds ~5.3G for the whole run; with
+ # it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime
+ # config, so they cannot affect the measured pass/fail signal.
+ # * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command
+ # below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends
+ # so our -Xmx2G comes last and wins. Gluten offloads data to Velox
+ # off-heap (capped at 2g via spark.memory.offHeap.size in the patched
+ # DeltaSQLCommandTest), so the fork's heap need is modest. A larger
+ # fork heap pushed the cgroup peak past the ~16G OOM threshold and the
+ # kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G
+ # keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is
+ # analyzable.
+ # `-u target/test-reports` enables ScalaTest's JUnit XML reporter so
+ # every suite writes per-test results. Delta itself only configures
+ # the console reporter (-oDF), so without this we'd have no machine-
+ # readable results to gate on. The path is relative to the forked
+ # test JVM's working dir (Test / baseDirectory = spark/), i.e.
+ # delta/spark/target/test-reports/TEST-*.xml.
+ #
+ # We deliberately do NOT let an sbt non-zero exit (which fires on the
+ # MANY expected Delta-on-Gluten failures) fail this step directly.
+ # Instead the known-failures gate below decides pass/fail: the build
+ # is green when the only failures are ones already recorded in the
+ # baseline, and red on a genuine regression.
+ set +e
+ # --- hang watchdog ---------------------------------------------------
+ # Shard 2 (and occasionally others) hangs indefinitely after a suite's
+ # last test with no further output. ScalaTest's failAfter only wraps
+ # individual test BODIES, so a wedge in suite teardown/afterAll -- or in
+ # a non-interruptible native Velox/JNI call that ignores
+ # Thread.interrupt() -- has no timeout and stalls until the 350-min job
+ # limit with zero diagnostics. This watchdog dumps the forked test JVM's
+ # threads (to the job log, and to a file for the artifact) once the test
+ # output has been silent for too long, so the deadlock is diagnosable.
+ SBT_LOG="/tmp/sbt-spark-test-shard-${{ matrix.shard }}.log"
+ : > "$SBT_LOG"
+ rm -f /tmp/sbt-done
+ (
+ # CRITICAL: the step shell runs with `bash -eo pipefail`, which the
+ # subshell inherits. Without `set +e` here, ANY non-zero command --
+ # e.g. fork detection finding no match, or `kill`/`jps` returning
+ # non-zero -- silently kills this watchdog. That errexit kill (plus a
+ # /proc detection miss) once made the watchdog capture ZERO dumps. A
+ # diagnostic must never abort on a failed probe.
+ set +e +o pipefail
+ JSTACK="${JAVA_HOME}/bin/jstack"
+ JPS="${JAVA_HOME}/bin/jps"
+ silent_limit=900 # 15 min with no new test output => treat as hung
+ dumps=0
+ fork_pids() {
+ # The sbt test fork's main class is sbt.ForkMain. Prefer jps (reads
+ # the main class from hsperfdata, robust to sbt's @argfile launch);
+ # fall back to scanning /proc cmdline + @argfile.
+ "$JPS" -l 2>/dev/null | awk '/sbt\.ForkMain/ {print $1}'
+ local p cl arg
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] || continue
+ cl="$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)"
+ case "$cl" in *sbt.ForkMain*) echo "${p##*/}"; continue ;; esac
+ arg="$(printf '%s' "$cl" | tr ' ' '\n' | sed -n 's/^@//p' | head -1)"
+ [ -n "$arg" ] && [ -f "$arg" ] && grep -qa 'sbt\.ForkMain' "$arg" 2>/dev/null \
+ && echo "${p##*/}"
+ done
+ }
+ all_java_pids() {
+ "$JPS" -q 2>/dev/null
+ local p
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] && echo "${p##*/}"
+ done
+ }
+ echo "HANG WATCHDOG armed: dumps the test JVM after ${silent_limit}s of output silence"
+ hb=0
+ while [ ! -f /tmp/sbt-done ]; do
+ sleep 60
+ [ -f "$SBT_LOG" ] || continue
+ now=$(date +%s)
+ mtime=$(stat -c %Y "$SBT_LOG" 2>/dev/null || echo "$now")
+ silent=$(( now - mtime ))
+ # Per-minute memory profile: heap tuning proved the ~16G OOM peak is
+ # NATIVE-driven, so log which JVM (sbt launcher vs fork) actually grows
+ # toward it -- the last lines before a hang reveal the real hog to cut.
+ # Read /proc directly (no `ps` dependency in the minimal container).
+ memnow=$(awk '{printf "%.2fG",$1/1073741824}' /sys/fs/cgroup/memory.current 2>/dev/null)
+ jvmrss=""
+ for mp in $(all_java_pids 2>/dev/null | sort -un); do
+ r=$(awk '/^VmRSS:/{print $2}' "/proc/$mp/status" 2>/dev/null)
+ [ -n "$r" ] && jvmrss="$jvmrss $(( r / 1024 ))M(p$mp)"
+ done
+ echo "MEM cgroup=${memnow} JVMs=[${jvmrss# }]"
+ hb=$(( hb + 1 ))
+ # Heartbeat every ~5 min so we can SEE the watchdog is alive (and how
+ # long the test has been silent) without waiting for a hang.
+ [ $(( hb % 5 )) -eq 0 ] && echo "HANG WATCHDOG: alive; last test output ${silent}s ago"
+ if [ "$silent" -ge "$silent_limit" ] && [ "$dumps" -lt 3 ]; then
+ dumps=$(( dumps + 1 ))
+ pids="$(fork_pids | sort -un)"
+ # Safety net: if the fork JVM cannot be pinpointed, dump EVERY JVM.
+ [ -n "$pids" ] || pids="$(all_java_pids | sort -un)"
+ echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $pids))"
+ [ -n "$pids" ] || echo "HANG WATCHDOG: no java process found to dump"
+ for pid in $pids; do
+ # SIGQUIT makes the JVM print a full thread dump to its OWN stderr,
+ # which sbt relays into the test log via the SAME stream as test
+ # output -- so it lands in the job log even when a separately
+ # spawned jstack child's output would be buffered/lost. Also write
+ # jstack to a file for the per-shard artifact.
+ echo "----- SIGQUIT + jstack pid ${pid} -----"
+ kill -QUIT "$pid" 2>/dev/null || echo "HANG WATCHDOG: kill -QUIT failed for pid ${pid}"
+ timeout 120 "$JSTACK" -l "$pid" > "/tmp/threaddump-shard-${{ matrix.shard }}-${dumps}-${pid}.txt" 2>&1 \
+ || echo "HANG WATCHDOG: jstack failed/timed out for pid ${pid}"
+ done
+ echo "::endgroup::"
+ # The dump is now captured (job log via SIGQUIT + artifact via
+ # jstack file). A hung JVM otherwise stalls the whole shard until
+ # the 350-min job timeout AND keeps the job log frozen so the dump
+ # never becomes reachable. So KILL the wedged JVM(s): the suite
+ # fails fast (acceptable -- errors are expected; only an
+ # unrecoverable hang blocks CI), the job proceeds/ends, and the log
+ # + artifacts flush. Give SIGQUIT a moment to print first.
+ sleep 20
+ echo "HANG WATCHDOG: killing wedged JVM(s) to unblock the shard: $(printf '%s ' $pids)"
+ for pid in $pids; do kill -KILL "$pid" 2>/dev/null; done
+ fi
+ done
+ ) &
+ WATCHDOG_PID=$!
+
+ ./build/sbt \
+ -DsparkVersion=${{ steps.resolve.outputs.spark_version }} \
+ -v \
+ -J-XX:+UseG1GC -J-Xms512m -J-Xmx4G \
+ -J-XX:G1PeriodicGCInterval=10000 \
+ -J-XX:G1PeriodicGCSystemLoadThreshold=0 \
+ -J-XX:-G1PeriodicGCInvokesConcurrent \
+ -J-XX:MinHeapFreeRatio=5 -J-XX:MaxHeapFreeRatio=15 \
+ "++ ${DELTA_SCALA_VERSION}" \
+ 'set spark / Test / javaOptions ++= Seq("-Xmx2G", "-XX:+HeapDumpOnOutOfMemoryError", "-XX:HeapDumpPath=/tmp/")' \
+ 'set spark / Test / testOptions += Tests.Argument(TestFrameworks.ScalaTest, "-u", "target/test-reports")' \
+ "spark/test" 2>&1 | tee "$SBT_LOG"
+ SBT_EXIT=${PIPESTATUS[0]}
+ touch /tmp/sbt-done
+ kill "$WATCHDOG_PID" 2>/dev/null || true
+ set -e
+ echo "sbt spark/test exited with ${SBT_EXIT}"
+
+ # Memory forensics: a sudden forked-JVM death with no hs_err and no heap
+ # dump is almost always a kernel/cgroup OOM-kill (Velox off-heap + JVM
+ # heap exceeding the ~16G runner). Surface the cgroup peak + oom_kill
+ # count so we can confirm/measure it (cgroup v2 paths; best-effort).
+ ( echo "=== cgroup memory forensics (exit ${SBT_EXIT}) ==="
+ for f in /sys/fs/cgroup/memory.peak /sys/fs/cgroup/memory.max \
+ /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.events; do
+ [ -r "$f" ] && { echo "--- $f ---"; cat "$f"; }
+ done ) || true
+
+ # A compile/launch failure leaves no reports at all. In that case the
+ # gate would see zero failures and pass spuriously, so fail loudly.
+ REPORT_COUNT=$(find . -path '*/target/test-reports/*.xml' 2>/dev/null | wc -l || true)
+ echo "Found ${REPORT_COUNT} JUnit XML report file(s)."
+ if [ "${REPORT_COUNT}" -eq 0 ]; then
+ echo "::error::sbt produced no test reports (exit ${SBT_EXIT}) -- likely a compile or launch failure, not test failures."
+ exit 1
+ fi
+
+ # update_baseline=true -> SEED mode (record failures, never fail) so the
+ # baseline can be (re)generated. Otherwise ENFORCE against the baseline.
+ GATE_MODE=enforce
+ if [ "${{ steps.resolve.outputs.update_baseline }}" = "true" ]; then
+ GATE_MODE=seed
+ fi
+ mkdir -p "$GITHUB_WORKSPACE/gate-out"
+ python3 "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/compare-test-results.py" \
+ --mode "${GATE_MODE}" \
+ --reports-dir "$GITHUB_WORKSPACE/delta" \
+ --known-failures "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/known-failures.txt" \
+ --failures-out "$GITHUB_WORKSPACE/gate-out/failures-shard-${{ matrix.shard }}.txt" \
+ --ran-out "$GITHUB_WORKSPACE/gate-out/ran-shard-${{ matrix.shard }}.txt" \
+ --fail-on-fixed "${{ steps.resolve.outputs.fail_on_fixed }}"
+
+ - name: Compress heap dumps (if any)
+ if: ${{ failure() }}
+ run: |
+ set -euo pipefail
+ if compgen -G "/tmp/*.hprof" > /dev/null; then
+ echo "Found heap dump(s); compressing..."
+ ls -lh /tmp/*.hprof
+ # gzip is single-threaded and slow on multi-GB heaps but is
+ # always present in the centos image. Heap dumps compress ~10x.
+ gzip -1 /tmp/*.hprof
+ ls -lh /tmp/*.hprof.gz
+ else
+ echo "No heap dumps found in /tmp/."
+ fi
+
+ - name: Upload per-shard gate lists
+ if: always()
+ uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gate-lists-shard-${{ matrix.shard }}
+ path: gate-out/*.txt
+ if-no-files-found: warn
+
+ - name: Upload test reports
+ if: always()
+ uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-reports-shard-${{ matrix.shard }}
+ path: |
+ delta/**/target/test-reports/**/*.xml
+ delta/**/target/surefire-reports/**/*.xml
+ if-no-files-found: warn
+
+ - name: Upload hang watchdog thread dumps
+ if: always()
+ uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-threaddumps-shard-${{ matrix.shard }}
+ path: /tmp/threaddump-shard-${{ matrix.shard }}-*.txt
+ if-no-files-found: ignore
+
+ - name: Upload JVM crash logs and other failure artifacts
+ if: ${{ failure() }}
+ uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-failure-logs-shard-${{ matrix.shard }}
+ path: |
+ delta/**/target/*.log
+ delta/**/hs_err_pid*.log
+ delta/**/core.*
+ /tmp/*.hprof
+ /tmp/*.hprof.gz
+ if-no-files-found: ignore
+
+ # Merges every shard's failure/ran lists into a single, sorted, ready-to-commit
+ # known-failures.txt and reports global regressions / now-passing / stale
+ # entries. Runs even when some shards went red (if: always()) so the refreshed
+ # baseline artifact is always available -- this is what you download and commit
+ # to bootstrap or refresh the baseline (see util/delta-spark-ut/README.md).
+ delta-spark-aggregate:
+ needs: delta-spark-test
+ if: always()
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download per-shard gate lists
+ uses: actions/download-artifact@v4
+ continue-on-error: true
+ with:
+ pattern: delta-spark-ut-gate-lists-shard-*
+ path: gate-lists
+ merge-multiple: true
Review Comment:
Good catch — fixed in 60c818a7f . The aggregator now takes --expected-shards (wired to DELTA_NUM_SHARDS ): it counts shards that produced a complete failures-* / ran-* pair and, if fewer than expected are present (a shard that died before writing its gate lists, or an artifact that failed to download), it fails with exit 2 before writing the baseline — so an incomplete known-failures.txt that would silently shrink the baseline can never be produced. Requiring both files per shard also catches a partial download.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3533114245
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,208 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
Review Comment:
Context with prior "Done" thread 3494090796 : that flagged the destructive rm -rf "$DELTA_DIR" , which was resolved by removing rm -rf entirely (the git init + fetch rewrite). So the catastrophic mass-delete is already gone; the residual is milder (a bad DELTA_DIR would git init in an unintended dir and let cp / sed overwrite files).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3549484879
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,692 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. `spark_version` is the single source of truth for the
+ # Spark version: it drives the Gluten bundle profile (-Pspark-<v>), the bundle
+ # jar name, and Delta's -DsparkVersion, so the tests always run against a bundle
+ # built for the same Spark version (no separate value to keep in sync). Scala
+ # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from
+ # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a
+ # compatible delta_ref.
+ GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version }}
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version }}
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Every input has a default (workflow_call + workflow_dispatch), so they
+ # are always set; just surface them as step outputs for the steps below.
+ env:
+ DELTA_REF: ${{ inputs.delta_ref }}
+ SPARK_VERSION: ${{ inputs.spark_version }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline }}
+ FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
+
+ - name: Download Gluten bundle jar
+ uses: actions/download-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: gluten-bundle
+
+ - name: Install minimal tools
+ run: |
+ set -euo pipefail
+ # apache/gluten:centos-9-jdk17 already has java-17, git, tar, a POSIX
+ # shell, and curl-minimal (which provides the `curl` command sbt's
+ # launcher needs). Install the rest of what Delta's build/sbt and the
+ # tests may need. We deliberately do NOT install the full `curl`
+ # package -- it conflicts with the pre-installed curl-minimal.
+ yum install -y java-17-openjdk-devel which findutils gzip python3
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ git --version
+ curl --version | head -n 1
+
+ - name: Cache sbt / Ivy / Coursier
+ uses: actions/cache@v4
+ with:
+ path: |
+ /root/.sbt
+ /root/.ivy2
+ /root/.cache/coursier
+ # Intentionally NOT keyed by ${{ matrix.shard }} -- all shards share
+ # the same dependency tree, so a single shared cache (with parallel
+ # save races resolved by GH on a first-write-wins basis) gives the
+ # best storage / hit-rate tradeoff.
+ key: delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-${{ env.DELTA_SCALA_VERSION }}
+ restore-keys: |
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-
+
+ - name: Clone and patch Delta
+ run: |
+ set -euo pipefail
+ # `2>/dev/null ... || true` keeps a no-match `ls` from aborting the step
+ # under `set -o pipefail`, so the explicit check below emits a clear
+ # error instead of a generic "cannot access".
+ GLUTEN_JAR=$(ls "$GITHUB_WORKSPACE"/gluten-bundle/gluten-velox-bundle-spark*_*-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$GLUTEN_JAR" ] || [ ! -f "$GLUTEN_JAR" ]; then
+ echo "ERROR: No Gluten bundle jar found under $GITHUB_WORKSPACE/gluten-bundle/" >&2
+ ls -la "$GITHUB_WORKSPACE/gluten-bundle/" || true
+ exit 1
+ fi
+ echo "Using Gluten bundle: $GLUTEN_JAR"
+ bash "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/setup-delta.sh" \
+ "${{ steps.resolve.outputs.delta_ref }}" \
+ "$GITHUB_WORKSPACE/delta" \
+ "$GLUTEN_JAR" \
+ "$GITHUB_WORKSPACE"
+
+ - name: Run Delta spark module tests (shard ${{ matrix.shard }} / ${{ env.DELTA_NUM_SHARDS }})
+ env:
+ NUM_SHARDS: ${{ env.DELTA_NUM_SHARDS }}
+ TEST_PARALLELISM_COUNT: ${{ steps.resolve.outputs.test_parallelism }}
+ # Required by Delta to enable testing-only code paths
+ # (see delta build.sbt: "Test / envVars += DELTA_TESTING -> 1").
+ DELTA_TESTING: '1'
+ # JDK 17 + Gluten/Arrow/Netty requires extra --add-opens and the
+ # `io.netty.tryReflectionSetAccessible` system property; otherwise
+ # the forked test JVM fails with
+ # java.lang.UnsupportedOperationException: sun.misc.Unsafe or
+ # java.nio.DirectByteBuffer.<init>(long, int) not available
+ # as soon as Gluten's bundled Arrow allocator initializes Netty
+ # direct buffers. Delta's own `Test / javaOptions` (see
+ # project/CrossSparkVersions.scala `java17TestSettings`) sets the
+ # base add-opens but NOT the Netty property -- Delta's own tests
+ # don't load Arrow/Netty buffers in a way that triggers it.
+ #
+ # Use JAVA_TOOL_OPTIONS so the flags propagate to BOTH the sbt
+ # launcher JVM and the forked test JVM (sbt forks tests and the
+ # child inherits the parent's env). The set below mirrors
+ # `extraJavaTestArgs` from Gluten's own root pom.xml (the
+ # canonical Gluten test JVM flag set).
+ #
+ # NOTE: we deliberately do NOT put `-Xmx` here. JAVA_TOOL_OPTIONS
+ # is processed BEFORE the JVM command line, so Delta's explicit
+ # `-Xmx1024m` (set in build.sbt `Test / javaOptions`) would still
+ # win (last `-Xmx` wins). The forked-test-JVM heap is bumped via
+ # an sbt `set spark / Test / javaOptions ++= ...` command below,
+ # which APPENDS to Delta's own seq -- so our `-Xmx` lands AFTER
+ # `-Xmx1024m` and wins.
+ JAVA_TOOL_OPTIONS: >-
+ -XX:+IgnoreUnrecognizedVMOptions
+ --add-opens=java.base/java.lang=ALL-UNNAMED
+ --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
+ --add-opens=java.base/java.lang.reflect=ALL-UNNAMED
+ --add-opens=java.base/java.io=ALL-UNNAMED
+ --add-opens=java.base/java.net=ALL-UNNAMED
+ --add-opens=java.base/java.nio=ALL-UNNAMED
+ --add-opens=java.base/java.util=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
+ --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.cs=ALL-UNNAMED
+ --add-opens=java.base/sun.security.action=ALL-UNNAMED
+ --add-opens=java.base/sun.util.calendar=ALL-UNNAMED
+ -Djdk.reflect.useDirectMethodHandle=false
+ -Dio.netty.tryReflectionSetAccessible=true
+ -Dfile.encoding=UTF-8
Review Comment:
I see. Could we add a similar sbt configuration in a shared location to make it easier for developers to run the tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3669339006
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -101,6 +106,80 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR
+ # touches high-signal Delta paths -- the Delta integration code
+ # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own
+ # files -- or carries the `run-delta-ci` opt-in label. Changes to general
+ # Velox/core/native code can also affect Delta offload but are touched
+ # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml
+ # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage
+ # down. NOTE: the label is read from the event that triggered this run, so add
+ # it before/with a push; labeling an already-finished run needs a new push.
+ delta-changes:
+ runs-on: ubuntu-22.04
+ outputs:
+ run_delta: ${{ steps.filter.outputs.run_delta }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ - name: Detect Delta-relevant changes / opt-in label
+ id: filter
+ env:
+ HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+ run: |
+ set -euo pipefail
+ # Opt-in label forces the suite even with no Delta-relevant path change.
+ if [ "$HAS_LABEL" = "true" ]; then
+ echo "run-delta-ci label present -> running Delta suite"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
+ fi
+ # Fail open if we can't determine the PR range (e.g. a non-PR trigger):
+ # never silently skip coverage.
+ if [ -z "${BASE_SHA:-}" ] || [ -z "${HEAD_SHA:-}" ]; then
+ echo "no PR base/head sha -> running Delta suite (fail-open)"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
+ fi
+ BASE=$(git merge-base "$BASE_SHA" "$HEAD_SHA" 2>/dev/null || echo "$BASE_SHA")
+ echo "diff base=$BASE head=$HEAD_SHA"
+ # Fail open if the diff itself can't be computed (missing objects after a
+ # force-push race, an unfetched fork head, ...). Piping straight into
+ # `grep -q` inside an `if` would hide that: git's failure leaves grep with
+ # empty input, so the pipeline exits non-zero exactly as it does for "no
+ # match" -- and `set -e`/`pipefail` can't help, since a tested command is
+ # allowed to fail. Capture the diff first so the two cases stay distinct.
+ if ! CHANGED=$(git diff --name-only "$BASE" "$HEAD_SHA"); then
Review Comment:
This PR is a fork PR ( felipepessoto/gluten → apache/gluten ), and across 4/4 runs delta-changes printed Delta-relevant paths changed , never git diff failed . The fork head is reachable because refs/pull/N/merge 's second parent is the fork head commit, so checkout brings it into the local object store.
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -101,6 +106,80 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR
+ # touches high-signal Delta paths -- the Delta integration code
+ # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own
+ # files -- or carries the `run-delta-ci` opt-in label. Changes to general
+ # Velox/core/native code can also affect Delta offload but are touched
+ # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml
+ # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage
+ # down. NOTE: the label is read from the event that triggered this run, so add
+ # it before/with a push; labeling an already-finished run needs a new push.
+ delta-changes:
+ runs-on: ubuntu-22.04
+ outputs:
+ run_delta: ${{ steps.filter.outputs.run_delta }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ - name: Detect Delta-relevant changes / opt-in label
+ id: filter
+ env:
+ HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
Review Comment:
Invalid, duplicated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3661688269
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -101,6 +106,70 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR
+ # touches high-signal Delta paths -- the Delta integration code
+ # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own
+ # files -- or carries the `run-delta-ci` opt-in label. Changes to general
+ # Velox/core/native code can also affect Delta offload but are touched
+ # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml
+ # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage
+ # down. NOTE: the label is read from the event that triggered this run, so add
+ # it before/with a push; labeling an already-finished run needs a new push.
+ delta-changes:
+ runs-on: ubuntu-22.04
+ outputs:
+ run_delta: ${{ steps.filter.outputs.run_delta }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ - name: Detect Delta-relevant changes / opt-in label
+ id: filter
+ env:
+ HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+ run: |
+ set -euo pipefail
+ # Opt-in label forces the suite even with no Delta-relevant path change.
+ if [ "$HAS_LABEL" = "true" ]; then
+ echo "run-delta-ci label present -> running Delta suite"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
+ fi
+ # Fail open if we can't determine the PR range (e.g. a non-PR trigger):
+ # never silently skip coverage.
+ if [ -z "${BASE_SHA:-}" ] || [ -z "${HEAD_SHA:-}" ]; then
+ echo "no PR base/head sha -> running Delta suite (fail-open)"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
+ fi
+ BASE=$(git merge-base "$BASE_SHA" "$HEAD_SHA" 2>/dev/null || echo "$BASE_SHA")
+ echo "diff base=$BASE head=$HEAD_SHA"
+ # High-signal Delta paths only: the Delta integration code
+ # (backends-velox/src-delta*), the Delta module, and this pipeline's own
+ # files. A change to general Velox/core/native code can also affect Delta
+ # offload, but those are touched constantly; per-PR we skip them (the
+ # nightly full run + the `run-delta-ci` label are the safety nets) to
+ # keep GHA usage down.
+ if git diff --name-only "$BASE" "$HEAD_SHA" | grep -Eq \
+ '^(\.github/workflows/velox_backend_x86\.yml|\.github/workflows/delta_spark_ut\.yml|\.github/workflows/util/delta-spark-ut/|gluten-delta/|backends-velox/src-delta)'; then
+ echo "Delta-relevant paths changed -> running Delta suite"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"
+ else
+ echo "No Delta-relevant paths changed and no opt-in label -> skipping Delta suite"
+ echo "run_delta=false" >> "$GITHUB_OUTPUT"
+ fi
Review Comment:
Valid, and a good catch — the gate really did fail *closed*, which is the opposite of what the comment right above it promises. Fixed in 7618d6f0d.
Reproduced it first:
```console
$ git diff --name-only <badsha> <badsha> | grep -Eq '^gluten-delta/'
fatal: bad object deadbeefdeadbeefdeadbeefdeadbeefdeadbeef
run_delta=FALSE <-- silently skips Delta coverage
```
The reason it slips through is that git's failure leaves `grep` with empty input, so `grep -Eq` exits 1 — indistinguishable from "no Delta paths changed". `set -euo pipefail` doesn't help either, since a command used as an `if` condition is explicitly allowed to fail. Worth noting this is reachable: the `git merge-base ... || echo "$BASE_SHA"` fallback just above already anticipates the shas being unusable, and it feeds straight into this diff.
The fix is what you suggested — capture the diff, treat a git failure as fail-open, then grep the captured output:
```bash
if ! CHANGED=$(git diff --name-only "$BASE" "$HEAD_SHA"); then
echo "git diff failed -> running Delta suite (fail-open)"
echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
fi
if printf '%s\n' "$CHANGED" | grep -Eq '^(...)'; then
```
Verified against a real repo across 8 scenarios: the three git-error cases (bad base+head, valid base + bad head, empty shas) now all fail open, and the five matching cases are unchanged (`gluten-delta/` → true, `backends-velox/src-delta40` → true, docs-only → false, `cpp/velox` only → false, no changes → false). I ran the old logic through the same harness to confirm it fails exactly where the new one passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3531280126
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,467 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline is empty (not yet bootstrapped) the mode automatically
+ degrades to ``seed`` so the first run is never spuriously red.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard).
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import glob
+import os
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped) sets of (suite, test) tuples. A test is
+ 'failed' if its <testcase> has a <failure> or <error> child, 'skipped' if
+ it has a <skipped> child, otherwise 'passed'. Suite-level aborts (a
+ <testsuite> reporting errors/failures with no failing <testcase>) are
+ recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = (suite, name)
+ tags = _child_local_tags(tc)
+ if "failure" in tags or "error" in tags:
+ failed.add(key)
+ suite_has_failing_tc = True
+ elif "skipped" in tags:
+ skipped.add(key)
+ else:
+ passed.add(key)
+
+ # Suite-level abort: counters say something failed but no testcase
+ # carried the failure (the suite blew up in beforeAll/constructor).
+ # Record a
+ # synthetic entry so the regression is visible.
+ try:
+ errors = int(ts.get("errors", "0") or "0")
+ failures = int(ts.get("failures", "0") or "0")
+ except ValueError:
+ errors = failures = 0
+ if (errors + failures) > 0 and not suite_has_failing_tc:
+ failed.add((suite_name, SUITE_ABORTED))
+
+ if not parsed_any:
+ eprint(
+ "WARNING: no JUnit <testsuite> elements found under {}".format(reports_dir)
+ )
+
+ # A test can't be both passed and failed; failure wins. Skipped only counts
+ # if the test was not otherwise seen (e.g. retried).
+ passed -= failed
+ skipped -= failed
+ skipped -= passed
+ return passed, failed, skipped
+
+
+# --------------------------------------------------------------------------- #
+# Reporting helpers
+# --------------------------------------------------------------------------- #
+def _summary_sink():
+ """Return a writer that mirrors to GITHUB_STEP_SUMMARY when available."""
+ path = os.environ.get("GITHUB_STEP_SUMMARY")
+ handle = open(path, "a", encoding="utf-8") if path else None
+
+ def write(line=""):
+ print(line)
+ if handle:
+ handle.write(line + "\n")
+
+ return write, handle
+
+
+def _print_block(write, title, entries, limit=50):
+ write("")
+ write("### {} ({})".format(title, len(entries)))
+ if not entries:
+ return
+ write("")
+ write("```")
+ for i, (suite, test) in enumerate(sorted(entries)):
+ if i >= limit:
+ write("... and {} more".format(len(entries) - limit))
+ break
+ write(format_entry(suite, test))
+ write("```")
+
+
+# --------------------------------------------------------------------------- #
+# Modes
+# --------------------------------------------------------------------------- #
+def run_enforce(args):
+ baseline = load_entries(args.known_failures)
+ passed, failed, skipped = parse_reports(args.reports_dir)
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3661712267
##########
.github/workflows/util/delta-spark-ut/run-delta-tests.sh:
##########
@@ -0,0 +1,275 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Runs the Delta `spark` module tests for one shard under the Gluten bundle:
+# arms a hang watchdog (thread-dumps + kills a wedged fork), invokes sbt
+# spark/test with the tuned JVM/heap flags, prints cgroup memory forensics, and
+# then gates the results against the baseline (compare-test-results.py). Extracted
+# from delta_spark_ut.yml so the workflow step stays readable.
+#
+# Driven by environment (set by the workflow step / job):
+# SHARD_ID - this shard's id (matrix.shard)
+# SPARK_VERSION - Delta -DsparkVersion value
+# UPDATE_BASELINE - 'true' -> gate seed mode; else enforce
+# FAIL_ON_FIXED - passed through to the gate
+# DELTA_SCALA_VERSION, NUM_SHARDS, TEST_PARALLELISM_COUNT, DELTA_TESTING
+# - test env (see the workflow step's `env:` block)
+# GITHUB_WORKSPACE - repo root (holds the Delta clone + util scripts)
+#
+# JAVA_TOOL_OPTIONS is set by sourcing java-test-args.sh (below), not the caller.
+
+set -euo pipefail
+export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+export PATH=$JAVA_HOME/bin:$PATH
+# Gluten/JDK17 test JVM flags (--add-opens + Netty property), shared with local
+# dev runs. Sets JAVA_TOOL_OPTIONS so it reaches the sbt launcher + forked JVMs.
+# shellcheck source=./java-test-args.sh
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/java-test-args.sh"
+cd "$GITHUB_WORKSPACE/delta"
+chmod +x build/sbt
+# Only run the unified `spark` sbt project, NOT `sparkGroup/test` --
+# `sparkGroup` aggregates many other projects (sparkV2, contribs,
+# sharing, connect*, ...) that are out of scope for this pipeline.
+#
+# JVM heap layout -- two memory consumers on the ~16G runner:
+# * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to
+# return idle memory during the (long) test phase via G1 periodic GC
+# (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the
+# busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent
+# forces each periodic GC to a full STW collection) that uncommits to a
+# tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low
+# -Xms512m floor.
+# Without this the idle launcher holds ~5.3G for the whole run; with
+# it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime
+# config, so they cannot affect the measured pass/fail signal.
+# * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command
+# below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends
+# so our -Xmx2G comes last and wins. Gluten offloads data to Velox
+# off-heap (capped at 2g via spark.memory.offHeap.size in the patched
+# DeltaSQLCommandTest), so the fork's heap need is modest. A larger
+# fork heap pushed the cgroup peak past the ~16G OOM threshold and the
+# kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G
+# keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is
+# analyzable.
+# `-u target/test-reports` enables ScalaTest's JUnit XML reporter so
+# every suite writes per-test results. Delta itself only configures
+# the console reporter (-oDF), so without this we'd have no machine-
+# readable results to gate on. The path is relative to the forked
+# test JVM's working dir (Test / baseDirectory = spark/), i.e.
+# delta/spark/target/test-reports/TEST-*.xml.
+#
+# We deliberately do NOT let an sbt non-zero exit (which fires on the
+# MANY expected Delta-on-Gluten failures) fail this step directly.
+# Instead the known-failures gate below decides pass/fail: the build
+# is green when the only failures are ones already recorded in the
+# baseline, and red on a genuine regression.
+set +e
+# --- hang watchdog ---------------------------------------------------
+# Shard 2 (and occasionally others) hangs indefinitely after a suite's
+# last test with no further output. ScalaTest's failAfter only wraps
+# individual test BODIES, so a wedge in suite teardown/afterAll -- or in
+# a non-interruptible native Velox/JNI call that ignores
+# Thread.interrupt() -- has no timeout and stalls until the 350-min job
+# limit with zero diagnostics. This watchdog dumps the forked test JVM's
+# threads (to the job log, and to a file for the artifact) once the test
+# output has been silent for too long, so the deadlock is diagnosable.
+SBT_LOG="/tmp/sbt-spark-test-shard-${SHARD_ID}.log"
+# Marker the watchdog touches when it KILLS a wedged test fork. The killed fork's
+# running suite plus every suite queued behind it never run and never write a
+# report, and since we ignore sbt's exit code the gate would only judge the
+# suites that DID report -- so the main flow fails the shard when this exists.
+WATCHDOG_KILL_MARKER="/tmp/sbt-watchdog-killed-shard-${SHARD_ID}"
+: > "$SBT_LOG"
+rm -f /tmp/sbt-done "$WATCHDOG_KILL_MARKER"
Review Comment:
The hang watchdog completion marker uses a non-shard-specific path (/tmp/sbt-done). Since this script is parameterized by SHARD_ID (and already uses shard-specific log/kill marker names), a global marker can cause interference if multiple shards are run concurrently in the same environment (e.g., local parallel runs). Make the marker shard-specific here so each watchdog tracks only its own shard.
This issue also appears in the following locations of the same file:
- line 135
- line 222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3660625949
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,459 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib artifact it already built, so the expensive
+ # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a
+ # called workflow runs as part of the caller run), so the jobs below download it
+ # by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+ # Nightly full run against the latest default branch. The per-PR entry point
+ # (velox_backend_x86.yml) now runs the Delta suite only when a PR touches
+ # Delta-relevant paths (or carries the opt-in label), to save GHA minutes; this
+ # scheduled run keeps full coverage once a day so rarer regressions are still
+ # caught. It builds its own native lib (build-native-lib-centos-7 below) since
+ # there is no caller to provide one, and uses the workflow's default inputs.
+ schedule:
+ - cron: '0 5 * * *'
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
Review Comment:
Setting `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true` weakens the Actions runtime safety checks and can allow JavaScript actions to run on deprecated/insecure Node runtimes. If this is not strictly required, remove it; if it is required for a specific legacy action, prefer scoping it as narrowly as possible (job/step-level) and add a short comment explaining why it’s needed.
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,230 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# init + shallow fetch resolves a tag, branch OR commit SHA in a single path
+# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the
+# destructive `rm -rf "$DELTA_DIR"` it required. `--` terminates options so a
+# DELTA_REF starting with `-` can't be misread as a git flag (this script is
+# workflow_dispatch-runnable with a user-supplied ref).
+git init -q "$DELTA_DIR"
+git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git
+git -C "$DELTA_DIR" fetch -q --depth 1 origin -- "$DELTA_REF"
+git -C "$DELTA_DIR" checkout -q FETCH_HEAD
Review Comment:
This assumes `DELTA_DIR` does not already exist (or is empty). In local/dev reruns (or if a prior step partially created the directory), `git init` / `remote add` can fail and force manual cleanup. Consider adding a small preflight that either (a) deletes `DELTA_DIR` when it’s a non-git directory, or (b) reuses an existing clone by validating it’s a git repo with `origin` set and then fetching/checking out the requested ref.
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,769 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline file exists but is empty (not yet bootstrapped) the mode
+ automatically degrades to ``seed`` so the first run is never spuriously red.
+ A *missing* ``--known-failures`` file is treated as a configuration error
+ (the gate fails) so a mis-referenced path can't silently pass.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard). Pass ``--expected-shards N``
+ to fail when fewer than ``N`` shards contributed gate lists (a shard that
+ died before writing them), so an incomplete baseline is never produced.
+
+Flaky quarantine (``--flaky-tests``)
+ Some Delta-on-Gluten failures are non-deterministic (e.g. a native bug that
+ only triggers on certain runtime plans), so they are neither a stable pass
+ nor a stable failure and cannot live in the baseline: baselining them turns
+ the gate red on every run where they pass, and leaving them out turns it red
+ on every run where they fail. ``flaky-tests.txt`` quarantines them -- a
+ quarantined test never counts as a regression (when it fails) nor as
+ now-passing (when it passes), and is excluded from the regenerated baseline.
+ Its SUITE is an fnmatch glob (so one line covers a root-cause family across
+ generated suite variants); its TEST name is matched exactly.
+
+Flaky quarantine by error signature (``--flaky-error-patterns``)
+ When a failure is caused by a known nondeterministic bug that surfaces on a
+ *different test each run* (e.g. the native Delta DV bitmap row-index error),
+ matching by test name is whack-a-mole. ``flaky-error-patterns.txt`` instead
+ quarantines by root cause: each line is a regex matched against a failed
+ test's <failure>/<error> text, and any failure that matches is treated as
+ flaky regardless of which test it landed on.
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import fnmatch
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+
+class NoReportsError(RuntimeError):
+ """Raised when no JUnit <testsuite> elements are found under reports_dir."""
+
+
+class CorruptReportError(NoReportsError):
+ """Raised when an expected JUnit report file (TEST-*.xml) fails to parse.
+
+ Subclasses NoReportsError so the enforce/seed handler treats a truncated
+ report as a hard data error (exit 2) instead of silently dropping the
+ suite's results and letting the gate pass on partial data.
+ """
+
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def normalize_key(suite, test):
+ """Normalize a (suite, test) key parsed from JUnit XML to match baseline keys.
+
+ Baseline/flaky entries round-trip through write_entries (which collapses CR/LF
+ in the test name to spaces) and parse_entry (which strips the whole
+ ``suite#test`` line). A raw XML name carrying a trailing newline or
+ surrounding whitespace would therefore never equal its normalized baseline
+ entry: the gate would keep reporting it as a REGRESSION, and the
+ copy-pasteable line it prints could never suppress it (load strips it back).
+ Delta test names are freeform, so a version bump could introduce exactly that.
+ Applying the identical format+parse round-trip here keeps the two sides in
+ sync (and is a no-op for the normal, whitespace-free names).
+ """
+ safe_test = (test or "").replace("\r", " ").replace("\n", " ")
+ normalized = parse_entry(format_entry(suite or "", safe_test))
+ # parse_entry only returns None for a blank/comment line, which a real
+ # testcase key is not; fall back to a bare strip to keep this total.
+ if normalized is None:
+ return ((suite or "").strip(), safe_test.strip())
+ return normalized
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def make_is_flaky(flaky_entries):
+ """Build a predicate that matches a (suite, test) tuple against flaky entries.
+
+ A flaky entry quarantines a test whose failure is known to be non-deterministic
+ (see flaky-tests.txt). The entry's SUITE is treated as an fnmatch glob so a
+ single line can cover a root-cause family across generated suite variants
+ (e.g. ``*DVs*Suite`` matches every deletion-vector merge suite, ``*`` matches
+ any suite); the TEST name is matched exactly (test names are freeform and may
+ contain glob metacharacters, so they are never globbed).
+ """
+ exact = set()
+ globbed = []
+ for suite, test in flaky_entries:
+ if any(ch in suite for ch in "*?["):
+ globbed.append((suite, test))
+ else:
+ exact.add((suite, test))
+
+ def is_flaky(entry):
+ if entry in exact:
+ return True
+ suite, test = entry
+ for glob_suite, glob_test in globbed:
+ if test == glob_test and fnmatch.fnmatchcase(suite, glob_suite):
+ return True
+ return False
+
+ return is_flaky
+
+
+def load_patterns(path):
+ """Load flaky-error regex patterns from a file (one per line).
+
+ Blank lines and ``#`` comments are ignored. Each remaining line is compiled
+ as a case-sensitive regex. These match the FAILURE TEXT of a failed test
+ (its JUnit <failure>/<error> message + stack), so a test that fails with a
+ known-nondeterministic native error (e.g. the Delta DV bitmap row-index bug)
+ can be quarantined by root cause instead of by exact test name.
+ """
+ patterns = []
+ if not path or not os.path.exists(path):
+ return patterns
+ with open(path, encoding="utf-8") as fh:
+ for line in fh:
+ line = line.rstrip("\n")
+ if not line.strip() or line.lstrip().startswith("#"):
+ continue
+ patterns.append(re.compile(line))
+ return patterns
+
+
+def make_signature_matcher(patterns):
+ """Return a predicate matching a failure text against any flaky-error pattern."""
+
+ def matches(text):
+ if not text:
+ return False
+ return any(p.search(text) for p in patterns)
+
+ return matches
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def _failure_text(tc):
+ """Concatenate the message attribute + body text of a testcase's
+ <failure>/<error> children, for error-signature matching."""
+ parts = []
+ for c in tc:
+ if c.tag.split("}")[-1] in ("failure", "error"):
+ msg = c.get("message")
+ if msg:
+ parts.append(msg)
+ if c.text:
+ parts.append(c.text)
+ return "\n".join(parts)
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped, fail_texts). The first three are sets of
+ (suite, test) tuples; fail_texts maps each failed (suite, test) to its
+ combined <failure>/<error> message + stack text (used for error-signature
+ quarantine). A test is 'failed' if its <testcase> has a <failure> or <error>
+ child, 'skipped' if it has a <skipped> child, otherwise 'passed'. Suite-level
+ aborts (a <testsuite> reporting errors/failures with no failing <testcase>)
+ are recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+ fail_texts = {}
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ # A TEST-*.xml that fails to parse is almost always a report truncated
+ # when a forked test JVM was killed mid-write (e.g. OOM). Silently
+ # skipping it drops that suite's results and could let the gate go
+ # green on partial data, so fail hard for report files. Other XML that
+ # merely matched the broad `target/**` glob is still skipped.
+ if os.path.basename(xml_file).startswith("TEST-"):
+ raise CorruptReportError(
+ "corrupt or truncated JUnit report {}: {}. Refusing to "
+ "evaluate the gate on partial data.".format(xml_file, exc)
+ )
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = normalize_key(suite, name)
+ tags = _child_local_tags(tc)
+ if "failure" in tags or "error" in tags:
+ failed.add(key)
+ suite_has_failing_tc = True
+ fail_texts[key] = _failure_text(tc)
+ elif "skipped" in tags:
+ skipped.add(key)
+ else:
+ passed.add(key)
+
+ # Suite-level abort: counters say something failed but no testcase
+ # carried the failure (the suite blew up in beforeAll/constructor).
+ # Record a
+ # synthetic entry so the regression is visible.
+ try:
+ errors = int(ts.get("errors", "0") or "0")
+ failures = int(ts.get("failures", "0") or "0")
+ except ValueError:
+ errors = failures = 0
+ if (errors + failures) > 0 and not suite_has_failing_tc:
+ failed.add(normalize_key(suite_name, SUITE_ABORTED))
+
+ if not parsed_any:
+ raise NoReportsError(
+ "No JUnit <testsuite> elements found under {}. The test reports are "
+ "missing or in an unexpected format -- refusing to evaluate the gate "
+ "on an empty result set (this would otherwise pass silently).".format(
+ reports_dir
+ )
+ )
+
+ # A test can't be both passed and failed; failure wins. Skipped only counts
+ # if the test was not otherwise seen (e.g. retried).
+ passed -= failed
+ skipped -= failed
+ skipped -= passed
+ return passed, failed, skipped, fail_texts
+
+
+# --------------------------------------------------------------------------- #
+# Reporting helpers
+# --------------------------------------------------------------------------- #
+def _summary_sink():
+ """Return a writer that mirrors to GITHUB_STEP_SUMMARY when available."""
+ path = os.environ.get("GITHUB_STEP_SUMMARY")
+ handle = open(path, "a", encoding="utf-8") if path else None
+
+ def write(line=""):
+ print(line)
+ if handle:
+ handle.write(line + "\n")
+
+ return write, handle
+
+
+def _print_block(write, title, entries, limit=50):
+ write("")
+ write("### {} ({})".format(title, len(entries)))
+ if not entries:
+ return
+ write("")
+ write("```")
+ for i, (suite, test) in enumerate(sorted(entries)):
+ if i >= limit:
+ write("... and {} more".format(len(entries) - limit))
+ break
+ write(format_entry(suite, test))
+ write("```")
+
+
+# --------------------------------------------------------------------------- #
+# Modes
+# --------------------------------------------------------------------------- #
+def run_enforce(args):
+ # In enforce mode a missing baseline file would make load_entries() return an
+ # empty set, silently degrading to seed mode and passing the gate without
+ # enforcing anything. Treat a missing path as a configuration error; an
+ # existing-but-empty file is still allowed (it legitimately seeds).
+ if args.mode == "enforce" and (
+ not args.known_failures or not os.path.exists(args.known_failures)
+ ):
+ eprint(
+ "ERROR: --known-failures '{}' does not exist. In enforce mode the "
+ "baseline file must exist (an existing-but-empty file is allowed and "
+ "triggers seed mode). Refusing to silently pass.".format(
+ args.known_failures
+ )
+ )
+ return 2
+ baseline = load_entries(args.known_failures)
+ name_flaky = make_is_flaky(load_entries(args.flaky_tests))
+ sig_matches = make_signature_matcher(load_patterns(args.flaky_error_patterns))
+ try:
+ passed, failed, skipped, fail_texts = parse_reports(args.reports_dir)
+ except NoReportsError as exc:
+ eprint("ERROR: {}".format(exc))
+ return 2
+
+ # A test is quarantined if its NAME is in flaky-tests.txt, or its failure
+ # TEXT matches a flaky-error signature (e.g. the native DV bitmap row-index
+ # bug that hits a different DV-merge test each run).
+ def sig_flaky(e):
+ return e in fail_texts and sig_matches(fail_texts[e])
+
+ def flaky_is(e):
+ return name_flaky(e) or sig_flaky(e)
+
+ # Always emit this shard's artifacts for the aggregation job. Signature-flaky
+ # failures are dropped from failures-out: the aggregate job works off these
+ # text-less lists and cannot re-derive the signature match, so excluding them
+ # here keeps the regenerated baseline from absorbing a flaky failure. Name-
+ # flaky entries stay (the aggregate re-filters them via flaky-tests.txt).
+ if args.failures_out:
+ write_entries(args.failures_out, {e for e in failed if not sig_flaky(e)})
+ if args.ran_out:
+ write_entries(args.ran_out, passed | failed)
Review Comment:
`--ran-out` currently excludes skipped tests, but aggregate mode uses `union_ran` to detect stale baseline entries (`stale = baseline - union_ran`). If a test is present in the JUnit XML but reported as `<skipped>`, it will be incorrectly treated as 'stale / not seen this run'. Include `skipped` in the `ran-out` set (and update the aggregate/stale semantics accordingly) so 'stale' means truly absent, not merely skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3660828919
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,238 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# init + shallow fetch resolves a tag, branch OR commit SHA in a single path
+# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the
+# destructive `rm -rf "$DELTA_DIR"` it required. `--` terminates options so a
+# DELTA_REF starting with `-` can't be misread as a git flag (this script is
+# workflow_dispatch-runnable with a user-supplied ref).
+#
+# Every step here is idempotent so a local re-run (or a CI re-run on a runner
+# that kept the workspace) resumes instead of dying: `git init` re-initializes
+# an existing repo harmlessly, but `remote add` errors out when `origin` already
+# exists, so drop it first; and `checkout -f` discards leftovers from a previous
+# partial run. Nothing worth keeping exists here yet -- the bundle jar and the
+# source patches below are applied *after* this block.
+git init -q "$DELTA_DIR"
+git -C "$DELTA_DIR" remote remove origin 2>/dev/null || true
+git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git
+git -C "$DELTA_DIR" fetch -q --depth 1 origin -- "$DELTA_REF"
+git -C "$DELTA_DIR" checkout -qf FETCH_HEAD
+git -C "$DELTA_DIR" --no-pager log -1 --oneline
+echo "::endgroup::"
+
+echo "::group::Injecting Gluten bundle jar onto the spark project's TEST classpath"
+# The Gluten bundle jar must be on the spark project's TEST runtime classpath
+# (so DeltaSQLCommandTest can load org.apache.gluten.GlutenPlugin by name) but
+# NOT on the COMPILE classpath of `sparkV1`, which is the project that holds
+# Delta's main sources. The bundle's transitive contents include extra symbols
+# under `org.apache.spark.sql` that collide with Delta's main sources -- e.g.
+# MergeOutputGeneration.scala imports both `org.apache.spark.sql._` and
+# `org.apache.spark.sql.delta.ClassicColumnConversions._`, and would then fail
+# with `reference to expression is ambiguous`.
+#
+# sbt auto-scans `<baseDirectory>/lib` via `unmanagedBase`. Two relevant
+# projects in Delta v4.2.0 have a `lib/` baseDirectory:
+# - sparkV1: `project in file("spark")` -> spark/lib
+# - spark : `project in file("spark-unified")` -> spark-unified/lib
+# unmanagedJars are project-scoped (NOT inherited by dependents), so dropping
+# the bundle into spark-unified/lib/ adds it to the unified `spark` project's
+# Compile *and* Test classpaths -- but NOT to sparkV1's. That's exactly what
+# we want:
+# * sparkV1/Compile sees ONLY Delta's regular deps -> Delta main compiles.
+# * spark/Test/fullClasspath sees the bundle -> tests load GlutenPlugin.
+# (Verified empirically: with bundle only in spark-unified/lib/, sbt's
+# `show sparkV1/Compile/dependencyClasspath` excludes the bundle and
+# `show spark/Test/fullClasspath` includes it.)
+#
+# We deliberately do NOT also drop the bundle into spark/lib/, which is what
+# caused the previous compile failure: spark/lib/ is sparkV1's unmanagedBase,
+# and putting the bundle there would re-introduce the ambiguity errors.
+SPARK_UNIFIED_LIB="$DELTA_DIR/spark-unified/lib"
+mkdir -p "$SPARK_UNIFIED_LIB"
+cp "$GLUTEN_BUNDLE_JAR" "$SPARK_UNIFIED_LIB/gluten-velox-bundle.jar"
+ls -lh "$SPARK_UNIFIED_LIB"
+echo "::endgroup::"
+
+echo "::group::Patching DeltaSQLCommandTest to enable Gluten plugin"
+TARGET="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$TARGET" ]; then
+ echo "Expected file not found in Delta clone: $TARGET" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
+ exit 1
+fi
+cp "$PATCH_SOURCE" "$TARGET"
+echo "Patched $TARGET"
+echo "--- diff vs. upstream ---"
+git -C "$DELTA_DIR" --no-pager diff -- "spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala" || true
+echo "::endgroup::"
+
+# Delta's tests collect file-source scans by matching the concrete
+# `FileSourceScanExec` case class; Gluten offloads the scan to
+# DeltaScanTransformer, a `FileSourceScanLike` sibling, so those matches miss
+# (`scala.MatchError: List()`, empty partition filters, broken column-pruning /
+# scan-metric checks across many suites). delta-io/delta#7104 and #7105 widen the
+# matches to the shared `FileSourceScanLike` interface that both the vanilla and
+# Gluten scans implement (behavior-preserving for vanilla). Both are merged
+# upstream but land after the pinned DELTA_REF (v4.2.0), so apply them here; once
+# DELTA_REF includes a fix, cherry_pick_delta_fix detects it and skips (see below).
+#
+# Depth-2 fetch brings each fix commit and its parent, which cherry-pick needs to
+# diff against (a depth-1 fetch grafts the parent away); `-n` stages the change
+# without requiring a committer identity.
+cherry_pick_delta_fix() {
+ local sha="$1" pr="$2"
+ git -C "$DELTA_DIR" fetch --quiet --depth 2 origin "$sha"
+ echo "Cherry-picking delta-io/delta${pr}"
+ if git -C "$DELTA_DIR" cherry-pick -n "$sha"; then
+ return 0
+ fi
+ # The cherry-pick did not apply. The usual cause is that the pinned DELTA_REF
+ # already contains this fix (e.g. after a version bump), which makes the patch
+ # empty/conflicting and would -- under `set -e` -- abort the whole setup. We
+ # can't use ancestry to tell "already contained" from a genuine conflict here
+ # (the clone is shallow, so `merge-base --is-ancestor` can't see past the graft),
+ # so recover the exact paths this fix touches -- leaving other setup such as the
+ # DeltaSQLCommandTest patch intact -- and continue. This is self-correcting: if
+ # the fix is genuinely still needed, the FileSourceScanLike failures it prevents
+ # resurface as gate regressions rather than being hidden by a hard abort here.
+ echo "Cherry-pick of delta-io/delta${pr} did not apply cleanly" \
+ "(most likely already contained in ${DELTA_REF}); skipping it."
+ local f
+ while IFS= read -r f; do
+ [ -n "$f" ] || continue
+ git -C "$DELTA_DIR" reset -q -- "$f" 2>/dev/null || true
+ git -C "$DELTA_DIR" checkout -q -- "$f" 2>/dev/null || true
+ done < <(git -C "$DELTA_DIR" diff-tree --no-commit-id --name-only -r "$sha")
+ # Clear any leftover sequencer state (harmless if none exists).
+ git -C "$DELTA_DIR" cherry-pick --quit 2>/dev/null || true
+ return 0
+}
+
+echo "::group::Cherry-picking upstream Delta FileSourceScanLike test fixes"
+cherry_pick_delta_fix 46bd45d57eadd7e528002a0ae7bd36ce5a456eca "#7104 (ScanReportHelper.collectScans)"
+cherry_pick_delta_fix 959e00e15f41f56afc1c9bb95d160c55c6dc7068 "#7105 (9 more test suites)"
+echo "::endgroup::"
+
+echo "::group::Force-failing memory-hog DeletionVectorsSuite 2B-row tests"
+# Two DeletionVectorsSuite tests read from / delete from a 2-billion-row table.
+# Under the Gluten Velox bundle they balloon the forked test JVM to ~13G of
+# NATIVE memory (row-index materialization) and the kernel/cgroup OOM-kills it.
+# The dead fork then wedges sbt, hanging the whole shard until the workflow's
+# hang-watchdog dumps threads and kills it (~16 min wasted, and every suite
+# QUEUED AFTER it in that fork is skipped) -- see delta_spark_ut.yml.
+#
+# Rather than silently `ignore` these (easy to forget), we make them FAIL FAST
+# with a clear message: the gap stays visible in the test reports / baseline
+# until the native memory blow-up is fixed, at which point this patch should be
+# removed. NOTE: making the suite complete also un-skips the rest of the shard's
+# suite queue, so the known-failures baseline must be refreshed after this.
+#
+# ORDER MATTERS: keep this sed AFTER the cherry-picks above. #7105 also edits
+# DeletionVectorsSuite.scala, and git cherry-pick aborts (exit 128) when the work
+# tree has uncommitted edits to a file it touches.
+DVS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala"
+if [ ! -f "$DVS" ]; then
+ echo "Expected file not found in Delta clone: $DVS" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed." >&2
+ exit 1
+fi
+# Inject `fail(...)` as the first statement of each test body (the line ending
+# in `) {`). Delta sets no -Xfatal-warnings / dead-code warning, so the now-
+# unreachable original body compiles fine. Keep each injected line <100 chars:
+# Delta's scalastyle enforces a 100-char line length on test sources. The full
+# rationale lives in this comment, so the in-test message stays terse.
+sed -i 's#huge table: read from tables of 2B rows with existing DV of many zeros") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV read OOMs the test JVM; see setup-delta.sh")#' "$DVS"
+sed -i 's#number of rows from tables of 2B rows with DVs") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV delete OOMs the test JVM; see setup-delta.sh")#' "$DVS"
+INJECTED=$(grep -c "Gluten CI] Force-failed" "$DVS" || true)
+if [ "$INJECTED" -ne 2 ]; then
Review Comment:
This force-fail injection is not idempotent: re-running `setup-delta.sh` in an existing workspace will inject duplicate `fail(...)` lines, causing `INJECTED` to exceed 2 and the script to abort. To keep reruns reliable (especially for local dev / CI reruns reusing workspace), reset `DeletionVectorsSuite.scala` to upstream before applying the `sed` (e.g., `git checkout -- \"$DVS\"`) or guard each injection by checking whether the fail line is already present before inserting.
##########
.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt:
##########
@@ -0,0 +1,43 @@
+# Flaky-error signatures for the Delta Spark UT (Gluten) gate.
+#
+# Each non-comment line is a Python regex matched (re.search, case-sensitive)
+# against a failed test's JUnit <failure>/<error> text (message + stack). A
+# failure that matches is QUARANTINED by root cause: it never counts as a
+# regression, and is dropped from the shard's failures list so it can't leak
+# into the regenerated baseline -- regardless of WHICH test it landed on.
+#
+# Use this (instead of flaky-tests.txt) when a known nondeterministic bug
+# surfaces on a different test each run, so matching by test name is
+# whack-a-mole. Prefer fixing the underlying bug and REMOVING the entry.
+#
+# ---------------------------------------------------------------------------
+# Native Delta bitmap aggregator receives an INVALID row index during a MERGE
+# that writes deletion vectors, and aborts. The garbage row index trips one of
+# two bounds checks, so the same root cause shows up with two messages:
+#
+# too large (Long.MAX_VALUE):
+# VeloxRuntimeError INVALID_STATE
+# Reason: Delta RoaringBitmapArray row index 9223372036854775807 exceeds max
+# representable value 9223372030412324864
+# Expression: value <= kMaxRepresentableValue
+# Function: addSafe File: .../velox/compute/delta/RoaringBitmapArray.cpp:92
+#
+# negative (garbage):
+# VeloxRuntimeError INVALID_STATE
+# Reason: Delta bitmap row index cannot be negative: -6254810385378525259
+# Expression: value >= 0
+# Function: addRowIndex File: .../operators/functions/delta/DeltaBitmapAggregator.cc:44
+#
+# Both are the same root cause (garbage row index into the DV bitmap aggregator)
+# but distinct native errors, so each has its own explicit pattern below --
+# deliberately specific (bound to the exact error string) rather than a broad
+# "Delta bitmap row index" match, to avoid masking an unrelated failure. Add a
+# new line if a further bounds-check variant appears. Intermittent (depends on
+# the runtime plan/scan/scheduling), so it hits a different *DVs*Suite MERGE test
+# on each run. Remove these once the row-index materialization is fixed in the
+# native backend. Tracked upstream (DV bitmap invalid row index).
+# ---------------------------------------------------------------------------
+# too-large (Long.MAX_VALUE) -- RoaringBitmapArray.cpp addSafe, value <= kMaxRepresentableValue
+Delta RoaringBitmapArray row index \d+ exceeds max representable value
+# negative garbage -- DeltaBitmapAggregator.cc addRowIndex, value >= 0
+Delta bitmap row index cannot be negative: -?\d+
Review Comment:
This regex is slightly broader than necessary: `-?\\d+` would also match a (hypothetical) positive index even though the message says 'cannot be negative'. Tightening it to `-\\d+` reduces the chance of accidentally quarantining an unrelated failure if the error message format changes.
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,459 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib artifact it already built, so the expensive
+ # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a
+ # called workflow runs as part of the caller run), so the jobs below download it
+ # by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+ # Nightly full run against the latest default branch. The per-PR entry point
+ # (velox_backend_x86.yml) now runs the Delta suite only when a PR touches
+ # Delta-relevant paths (or carries the opt-in label), to save GHA minutes; this
+ # scheduled run keeps full coverage once a day so rarer regressions are still
+ # caught. It builds its own native lib (build-native-lib-centos-7 below) since
+ # there is no caller to provide one, and uses the workflow's default inputs.
+ schedule:
+ - cron: '0 5 * * *'
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
Review Comment:
`ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true` relaxes GitHub Actions’ Node runtime safety checks and can allow older EOL Node versions for actions, which is a supply-chain/security risk. If this isn’t strictly required for a specific legacy action, remove it; if it is required, scope it as narrowly as possible (job/step) and document which action requires it and why.
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -101,6 +106,70 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR
+ # touches high-signal Delta paths -- the Delta integration code
+ # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own
+ # files -- or carries the `run-delta-ci` opt-in label. Changes to general
+ # Velox/core/native code can also affect Delta offload but are touched
+ # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml
+ # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage
+ # down. NOTE: the label is read from the event that triggered this run, so add
+ # it before/with a push; labeling an already-finished run needs a new push.
+ delta-changes:
+ runs-on: ubuntu-22.04
+ outputs:
+ run_delta: ${{ steps.filter.outputs.run_delta }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ - name: Detect Delta-relevant changes / opt-in label
+ id: filter
+ env:
+ HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
Review Comment:
The expression `github.event.pull_request.labels.*.name` can fail to evaluate on non-PR events (e.g., `workflow_dispatch`, `push`) because `github.event.pull_request` is absent. This would break the gating job before your bash-level fail-open logic runs. Guard the expression with an event check (e.g., only evaluate `contains(...)` when `github.event_name == 'pull_request'`) or use a conditional expression that safely falls back when `pull_request` is missing.
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,820 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline file exists but is empty (not yet bootstrapped) the mode
+ automatically degrades to ``seed`` so the first run is never spuriously red.
+ A *missing* ``--known-failures`` file is treated as a configuration error
+ (the gate fails) so a mis-referenced path can't silently pass.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` / ``--skipped-out``
+ file into a single, sorted, ready-to-commit ``known-failures.txt`` and report
+ stale baseline entries (tests no longer present in any shard). Skipped tests
+ are tracked apart from run tests so "stale" means *truly absent* rather than
+ merely skipped this run; they are also kept out of the now-passing set, since
+ a skipped test is not evidence of a fix. Pass ``--expected-shards N``
+ to fail when fewer than ``N`` shards contributed gate lists (a shard that
+ died before writing them), so an incomplete baseline is never produced.
+
+Flaky quarantine (``--flaky-tests``)
+ Some Delta-on-Gluten failures are non-deterministic (e.g. a native bug that
+ only triggers on certain runtime plans), so they are neither a stable pass
+ nor a stable failure and cannot live in the baseline: baselining them turns
+ the gate red on every run where they pass, and leaving them out turns it red
+ on every run where they fail. ``flaky-tests.txt`` quarantines them -- a
+ quarantined test never counts as a regression (when it fails) nor as
+ now-passing (when it passes), and is excluded from the regenerated baseline.
+ Its SUITE is an fnmatch glob (so one line covers a root-cause family across
+ generated suite variants); its TEST name is matched exactly.
+
+Flaky quarantine by error signature (``--flaky-error-patterns``)
+ When a failure is caused by a known nondeterministic bug that surfaces on a
+ *different test each run* (e.g. the native Delta DV bitmap row-index error),
+ matching by test name is whack-a-mole. ``flaky-error-patterns.txt`` instead
+ quarantines by root cause: each line is a regex matched against a failed
+ test's <failure>/<error> text, and any failure that matches is treated as
+ flaky regardless of which test it landed on.
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import fnmatch
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+
+class NoReportsError(RuntimeError):
+ """Raised when no JUnit <testsuite> elements are found under reports_dir."""
+
+
+class CorruptReportError(NoReportsError):
+ """Raised when an expected JUnit report file (TEST-*.xml) fails to parse.
+
+ Subclasses NoReportsError so the enforce/seed handler treats a truncated
+ report as a hard data error (exit 2) instead of silently dropping the
+ suite's results and letting the gate pass on partial data.
+ """
+
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def normalize_key(suite, test):
+ """Normalize a (suite, test) key parsed from JUnit XML to match baseline keys.
+
+ Baseline/flaky entries round-trip through write_entries (which collapses CR/LF
+ in the test name to spaces) and parse_entry (which strips the whole
+ ``suite#test`` line). A raw XML name carrying a trailing newline or
+ surrounding whitespace would therefore never equal its normalized baseline
+ entry: the gate would keep reporting it as a REGRESSION, and the
+ copy-pasteable line it prints could never suppress it (load strips it back).
+ Delta test names are freeform, so a version bump could introduce exactly that.
+ Applying the identical format+parse round-trip here keeps the two sides in
+ sync (and is a no-op for the normal, whitespace-free names).
+ """
+ safe_test = (test or "").replace("\r", " ").replace("\n", " ")
+ normalized = parse_entry(format_entry(suite or "", safe_test))
+ # parse_entry only returns None for a blank/comment line, which a real
+ # testcase key is not; fall back to a bare strip to keep this total.
+ if normalized is None:
+ return ((suite or "").strip(), safe_test.strip())
+ return normalized
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def make_is_flaky(flaky_entries):
+ """Build a predicate that matches a (suite, test) tuple against flaky entries.
+
+ A flaky entry quarantines a test whose failure is known to be non-deterministic
+ (see flaky-tests.txt). The entry's SUITE is treated as an fnmatch glob so a
+ single line can cover a root-cause family across generated suite variants
+ (e.g. ``*DVs*Suite`` matches every deletion-vector merge suite, ``*`` matches
+ any suite); the TEST name is matched exactly (test names are freeform and may
+ contain glob metacharacters, so they are never globbed).
+ """
+ exact = set()
+ globbed = []
+ for suite, test in flaky_entries:
+ if any(ch in suite for ch in "*?["):
+ globbed.append((suite, test))
+ else:
+ exact.add((suite, test))
+
+ def is_flaky(entry):
+ if entry in exact:
+ return True
+ suite, test = entry
+ for glob_suite, glob_test in globbed:
+ if test == glob_test and fnmatch.fnmatchcase(suite, glob_suite):
+ return True
+ return False
+
+ return is_flaky
+
+
+def load_patterns(path):
+ """Load flaky-error regex patterns from a file (one per line).
+
+ Blank lines and ``#`` comments are ignored. Each remaining line is compiled
+ as a case-sensitive regex. These match the FAILURE TEXT of a failed test
+ (its JUnit <failure>/<error> message + stack), so a test that fails with a
+ known-nondeterministic native error (e.g. the Delta DV bitmap row-index bug)
+ can be quarantined by root cause instead of by exact test name.
+ """
+ patterns = []
+ if not path or not os.path.exists(path):
+ return patterns
+ with open(path, encoding="utf-8") as fh:
+ for line in fh:
+ line = line.rstrip("\n")
+ if not line.strip() or line.lstrip().startswith("#"):
+ continue
+ patterns.append(re.compile(line))
+ return patterns
Review Comment:
If `flaky-error-patterns.txt` contains an invalid regex, `re.compile(line)` will raise `re.error` and crash the script without a clear, actionable message about which line/pattern broke CI. Catch `re.error` here and emit a targeted error including the file path and offending pattern (and ideally its line number), then exit with a non-zero code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3549183322
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,692 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. `spark_version` is the single source of truth for the
+ # Spark version: it drives the Gluten bundle profile (-Pspark-<v>), the bundle
+ # jar name, and Delta's -DsparkVersion, so the tests always run against a bundle
+ # built for the same Spark version (no separate value to keep in sync). Scala
+ # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from
+ # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a
+ # compatible delta_ref.
+ GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version }}
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version }}
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Every input has a default (workflow_call + workflow_dispatch), so they
+ # are always set; just surface them as step outputs for the steps below.
+ env:
+ DELTA_REF: ${{ inputs.delta_ref }}
+ SPARK_VERSION: ${{ inputs.spark_version }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline }}
+ FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
+
+ - name: Download Gluten bundle jar
+ uses: actions/download-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: gluten-bundle
+
+ - name: Install minimal tools
+ run: |
+ set -euo pipefail
+ # apache/gluten:centos-9-jdk17 already has java-17, git, tar, a POSIX
+ # shell, and curl-minimal (which provides the `curl` command sbt's
+ # launcher needs). Install the rest of what Delta's build/sbt and the
+ # tests may need. We deliberately do NOT install the full `curl`
+ # package -- it conflicts with the pre-installed curl-minimal.
+ yum install -y java-17-openjdk-devel which findutils gzip python3
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ git --version
+ curl --version | head -n 1
+
+ - name: Cache sbt / Ivy / Coursier
+ uses: actions/cache@v4
+ with:
+ path: |
+ /root/.sbt
+ /root/.ivy2
+ /root/.cache/coursier
+ # Intentionally NOT keyed by ${{ matrix.shard }} -- all shards share
+ # the same dependency tree, so a single shared cache (with parallel
+ # save races resolved by GH on a first-write-wins basis) gives the
+ # best storage / hit-rate tradeoff.
+ key: delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-${{ env.DELTA_SCALA_VERSION }}
+ restore-keys: |
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-
+
+ - name: Clone and patch Delta
+ run: |
+ set -euo pipefail
+ # `2>/dev/null ... || true` keeps a no-match `ls` from aborting the step
+ # under `set -o pipefail`, so the explicit check below emits a clear
+ # error instead of a generic "cannot access".
+ GLUTEN_JAR=$(ls "$GITHUB_WORKSPACE"/gluten-bundle/gluten-velox-bundle-spark*_*-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$GLUTEN_JAR" ] || [ ! -f "$GLUTEN_JAR" ]; then
+ echo "ERROR: No Gluten bundle jar found under $GITHUB_WORKSPACE/gluten-bundle/" >&2
+ ls -la "$GITHUB_WORKSPACE/gluten-bundle/" || true
+ exit 1
+ fi
+ echo "Using Gluten bundle: $GLUTEN_JAR"
+ bash "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/setup-delta.sh" \
+ "${{ steps.resolve.outputs.delta_ref }}" \
+ "$GITHUB_WORKSPACE/delta" \
+ "$GLUTEN_JAR" \
+ "$GITHUB_WORKSPACE"
+
+ - name: Run Delta spark module tests (shard ${{ matrix.shard }} / ${{ env.DELTA_NUM_SHARDS }})
+ env:
+ NUM_SHARDS: ${{ env.DELTA_NUM_SHARDS }}
+ TEST_PARALLELISM_COUNT: ${{ steps.resolve.outputs.test_parallelism }}
+ # Required by Delta to enable testing-only code paths
+ # (see delta build.sbt: "Test / envVars += DELTA_TESTING -> 1").
+ DELTA_TESTING: '1'
+ # JDK 17 + Gluten/Arrow/Netty requires extra --add-opens and the
+ # `io.netty.tryReflectionSetAccessible` system property; otherwise
+ # the forked test JVM fails with
+ # java.lang.UnsupportedOperationException: sun.misc.Unsafe or
+ # java.nio.DirectByteBuffer.<init>(long, int) not available
+ # as soon as Gluten's bundled Arrow allocator initializes Netty
+ # direct buffers. Delta's own `Test / javaOptions` (see
+ # project/CrossSparkVersions.scala `java17TestSettings`) sets the
+ # base add-opens but NOT the Netty property -- Delta's own tests
+ # don't load Arrow/Netty buffers in a way that triggers it.
+ #
+ # Use JAVA_TOOL_OPTIONS so the flags propagate to BOTH the sbt
+ # launcher JVM and the forked test JVM (sbt forks tests and the
+ # child inherits the parent's env). The set below mirrors
+ # `extraJavaTestArgs` from Gluten's own root pom.xml (the
+ # canonical Gluten test JVM flag set).
+ #
+ # NOTE: we deliberately do NOT put `-Xmx` here. JAVA_TOOL_OPTIONS
+ # is processed BEFORE the JVM command line, so Delta's explicit
+ # `-Xmx1024m` (set in build.sbt `Test / javaOptions`) would still
+ # win (last `-Xmx` wins). The forked-test-JVM heap is bumped via
+ # an sbt `set spark / Test / javaOptions ++= ...` command below,
+ # which APPENDS to Delta's own seq -- so our `-Xmx` lands AFTER
+ # `-Xmx1024m` and wins.
+ JAVA_TOOL_OPTIONS: >-
+ -XX:+IgnoreUnrecognizedVMOptions
+ --add-opens=java.base/java.lang=ALL-UNNAMED
+ --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
+ --add-opens=java.base/java.lang.reflect=ALL-UNNAMED
+ --add-opens=java.base/java.io=ALL-UNNAMED
+ --add-opens=java.base/java.net=ALL-UNNAMED
+ --add-opens=java.base/java.nio=ALL-UNNAMED
+ --add-opens=java.base/java.util=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
+ --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.cs=ALL-UNNAMED
+ --add-opens=java.base/sun.security.action=ALL-UNNAMED
+ --add-opens=java.base/sun.util.calendar=ALL-UNNAMED
+ -Djdk.reflect.useDirectMethodHandle=false
+ -Dio.netty.tryReflectionSetAccessible=true
+ -Dfile.encoding=UTF-8
Review Comment:
The root pom has `extraJavaTestArgs` configured. Just wanted to confirm if we still need those setting here.
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,692 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. `spark_version` is the single source of truth for the
+ # Spark version: it drives the Gluten bundle profile (-Pspark-<v>), the bundle
+ # jar name, and Delta's -DsparkVersion, so the tests always run against a bundle
+ # built for the same Spark version (no separate value to keep in sync). Scala
+ # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from
+ # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a
+ # compatible delta_ref.
+ GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version }}
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version }}
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Every input has a default (workflow_call + workflow_dispatch), so they
+ # are always set; just surface them as step outputs for the steps below.
+ env:
+ DELTA_REF: ${{ inputs.delta_ref }}
+ SPARK_VERSION: ${{ inputs.spark_version }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline }}
+ FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
+
+ - name: Download Gluten bundle jar
+ uses: actions/download-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: gluten-bundle
+
+ - name: Install minimal tools
+ run: |
+ set -euo pipefail
+ # apache/gluten:centos-9-jdk17 already has java-17, git, tar, a POSIX
+ # shell, and curl-minimal (which provides the `curl` command sbt's
+ # launcher needs). Install the rest of what Delta's build/sbt and the
+ # tests may need. We deliberately do NOT install the full `curl`
+ # package -- it conflicts with the pre-installed curl-minimal.
+ yum install -y java-17-openjdk-devel which findutils gzip python3
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ git --version
+ curl --version | head -n 1
+
+ - name: Cache sbt / Ivy / Coursier
+ uses: actions/cache@v4
+ with:
+ path: |
+ /root/.sbt
+ /root/.ivy2
+ /root/.cache/coursier
+ # Intentionally NOT keyed by ${{ matrix.shard }} -- all shards share
+ # the same dependency tree, so a single shared cache (with parallel
+ # save races resolved by GH on a first-write-wins basis) gives the
+ # best storage / hit-rate tradeoff.
+ key: delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-${{ env.DELTA_SCALA_VERSION }}
+ restore-keys: |
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-
+
+ - name: Clone and patch Delta
+ run: |
+ set -euo pipefail
+ # `2>/dev/null ... || true` keeps a no-match `ls` from aborting the step
+ # under `set -o pipefail`, so the explicit check below emits a clear
+ # error instead of a generic "cannot access".
+ GLUTEN_JAR=$(ls "$GITHUB_WORKSPACE"/gluten-bundle/gluten-velox-bundle-spark*_*-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$GLUTEN_JAR" ] || [ ! -f "$GLUTEN_JAR" ]; then
+ echo "ERROR: No Gluten bundle jar found under $GITHUB_WORKSPACE/gluten-bundle/" >&2
+ ls -la "$GITHUB_WORKSPACE/gluten-bundle/" || true
+ exit 1
+ fi
+ echo "Using Gluten bundle: $GLUTEN_JAR"
+ bash "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/setup-delta.sh" \
+ "${{ steps.resolve.outputs.delta_ref }}" \
+ "$GITHUB_WORKSPACE/delta" \
+ "$GLUTEN_JAR" \
+ "$GITHUB_WORKSPACE"
+
+ - name: Run Delta spark module tests (shard ${{ matrix.shard }} / ${{ env.DELTA_NUM_SHARDS }})
+ env:
+ NUM_SHARDS: ${{ env.DELTA_NUM_SHARDS }}
+ TEST_PARALLELISM_COUNT: ${{ steps.resolve.outputs.test_parallelism }}
+ # Required by Delta to enable testing-only code paths
+ # (see delta build.sbt: "Test / envVars += DELTA_TESTING -> 1").
+ DELTA_TESTING: '1'
+ # JDK 17 + Gluten/Arrow/Netty requires extra --add-opens and the
+ # `io.netty.tryReflectionSetAccessible` system property; otherwise
+ # the forked test JVM fails with
+ # java.lang.UnsupportedOperationException: sun.misc.Unsafe or
+ # java.nio.DirectByteBuffer.<init>(long, int) not available
+ # as soon as Gluten's bundled Arrow allocator initializes Netty
+ # direct buffers. Delta's own `Test / javaOptions` (see
+ # project/CrossSparkVersions.scala `java17TestSettings`) sets the
+ # base add-opens but NOT the Netty property -- Delta's own tests
+ # don't load Arrow/Netty buffers in a way that triggers it.
+ #
+ # Use JAVA_TOOL_OPTIONS so the flags propagate to BOTH the sbt
+ # launcher JVM and the forked test JVM (sbt forks tests and the
+ # child inherits the parent's env). The set below mirrors
+ # `extraJavaTestArgs` from Gluten's own root pom.xml (the
+ # canonical Gluten test JVM flag set).
+ #
+ # NOTE: we deliberately do NOT put `-Xmx` here. JAVA_TOOL_OPTIONS
+ # is processed BEFORE the JVM command line, so Delta's explicit
+ # `-Xmx1024m` (set in build.sbt `Test / javaOptions`) would still
+ # win (last `-Xmx` wins). The forked-test-JVM heap is bumped via
+ # an sbt `set spark / Test / javaOptions ++= ...` command below,
+ # which APPENDS to Delta's own seq -- so our `-Xmx` lands AFTER
+ # `-Xmx1024m` and wins.
+ JAVA_TOOL_OPTIONS: >-
+ -XX:+IgnoreUnrecognizedVMOptions
+ --add-opens=java.base/java.lang=ALL-UNNAMED
+ --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
+ --add-opens=java.base/java.lang.reflect=ALL-UNNAMED
+ --add-opens=java.base/java.io=ALL-UNNAMED
+ --add-opens=java.base/java.net=ALL-UNNAMED
+ --add-opens=java.base/java.nio=ALL-UNNAMED
+ --add-opens=java.base/java.util=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
+ --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.cs=ALL-UNNAMED
+ --add-opens=java.base/sun.security.action=ALL-UNNAMED
+ --add-opens=java.base/sun.util.calendar=ALL-UNNAMED
+ -Djdk.reflect.useDirectMethodHandle=false
+ -Dio.netty.tryReflectionSetAccessible=true
+ -Dfile.encoding=UTF-8
+ run: |
+ set -euo pipefail
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ cd "$GITHUB_WORKSPACE/delta"
+ chmod +x build/sbt
+ # Only run the unified `spark` sbt project, NOT `sparkGroup/test` --
+ # `sparkGroup` aggregates many other projects (sparkV2, contribs,
+ # sharing, connect*, ...) that are out of scope for this pipeline.
+ #
+ # JVM heap layout -- two memory consumers on the ~16G runner:
+ # * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to
+ # return idle memory during the (long) test phase via G1 periodic GC
+ # (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the
+ # busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent
+ # forces each periodic GC to a full STW collection) that uncommits to a
+ # tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low
+ # -Xms512m floor.
+ # Without this the idle launcher holds ~5.3G for the whole run; with
+ # it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime
+ # config, so they cannot affect the measured pass/fail signal.
+ # * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command
+ # below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends
+ # so our -Xmx2G comes last and wins. Gluten offloads data to Velox
+ # off-heap (capped at 2g via spark.memory.offHeap.size in the patched
+ # DeltaSQLCommandTest), so the fork's heap need is modest. A larger
+ # fork heap pushed the cgroup peak past the ~16G OOM threshold and the
+ # kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G
+ # keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is
+ # analyzable.
+ # `-u target/test-reports` enables ScalaTest's JUnit XML reporter so
+ # every suite writes per-test results. Delta itself only configures
+ # the console reporter (-oDF), so without this we'd have no machine-
+ # readable results to gate on. The path is relative to the forked
+ # test JVM's working dir (Test / baseDirectory = spark/), i.e.
+ # delta/spark/target/test-reports/TEST-*.xml.
+ #
+ # We deliberately do NOT let an sbt non-zero exit (which fires on the
+ # MANY expected Delta-on-Gluten failures) fail this step directly.
+ # Instead the known-failures gate below decides pass/fail: the build
+ # is green when the only failures are ones already recorded in the
+ # baseline, and red on a genuine regression.
+ set +e
+ # --- hang watchdog ---------------------------------------------------
+ # Shard 2 (and occasionally others) hangs indefinitely after a suite's
+ # last test with no further output. ScalaTest's failAfter only wraps
+ # individual test BODIES, so a wedge in suite teardown/afterAll -- or in
+ # a non-interruptible native Velox/JNI call that ignores
+ # Thread.interrupt() -- has no timeout and stalls until the 350-min job
+ # limit with zero diagnostics. This watchdog dumps the forked test JVM's
+ # threads (to the job log, and to a file for the artifact) once the test
+ # output has been silent for too long, so the deadlock is diagnosable.
+ SBT_LOG="/tmp/sbt-spark-test-shard-${{ matrix.shard }}.log"
+ : > "$SBT_LOG"
+ rm -f /tmp/sbt-done
+ (
+ # CRITICAL: the step shell runs with `bash -eo pipefail`, which the
+ # subshell inherits. Without `set +e` here, ANY non-zero command --
+ # e.g. fork detection finding no match, or `kill`/`jps` returning
+ # non-zero -- silently kills this watchdog. That errexit kill (plus a
+ # /proc detection miss) once made the watchdog capture ZERO dumps. A
+ # diagnostic must never abort on a failed probe.
+ set +e +o pipefail
+ JSTACK="${JAVA_HOME}/bin/jstack"
+ JPS="${JAVA_HOME}/bin/jps"
+ silent_limit=900 # 15 min with no new test output => treat as hung
+ dumps=0
+ fork_pids() {
+ # The sbt test fork's main class is sbt.ForkMain. Prefer jps (reads
+ # the main class from hsperfdata, robust to sbt's @argfile launch);
+ # fall back to scanning /proc cmdline + @argfile.
+ "$JPS" -l 2>/dev/null | awk '/sbt\.ForkMain/ {print $1}'
+ local p cl arg
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] || continue
+ cl="$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)"
+ case "$cl" in *sbt.ForkMain*) echo "${p##*/}"; continue ;; esac
+ arg="$(printf '%s' "$cl" | tr ' ' '\n' | sed -n 's/^@//p' | head -1)"
+ [ -n "$arg" ] && [ -f "$arg" ] && grep -qa 'sbt\.ForkMain' "$arg" 2>/dev/null \
+ && echo "${p##*/}"
+ done
+ }
+ all_java_pids() {
+ "$JPS" -q 2>/dev/null
+ local p
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] && echo "${p##*/}"
+ done
+ }
+ echo "HANG WATCHDOG armed: dumps the test JVM after ${silent_limit}s of output silence"
+ hb=0
+ while [ ! -f /tmp/sbt-done ]; do
+ sleep 60
+ [ -f "$SBT_LOG" ] || continue
+ now=$(date +%s)
+ mtime=$(stat -c %Y "$SBT_LOG" 2>/dev/null || echo "$now")
+ silent=$(( now - mtime ))
+ # Per-minute memory profile: heap tuning proved the ~16G OOM peak is
+ # NATIVE-driven, so log which JVM (sbt launcher vs fork) actually grows
+ # toward it -- the last lines before a hang reveal the real hog to cut.
+ # Read /proc directly (no `ps` dependency in the minimal container).
+ memnow=$(awk '{printf "%.2fG",$1/1073741824}' /sys/fs/cgroup/memory.current 2>/dev/null)
+ jvmrss=""
+ for mp in $(all_java_pids 2>/dev/null | sort -un); do
+ r=$(awk '/^VmRSS:/{print $2}' "/proc/$mp/status" 2>/dev/null)
+ [ -n "$r" ] && jvmrss="$jvmrss $(( r / 1024 ))M(p$mp)"
+ done
+ echo "MEM cgroup=${memnow} JVMs=[${jvmrss# }]"
+ hb=$(( hb + 1 ))
+ # Heartbeat every ~5 min so we can SEE the watchdog is alive (and how
+ # long the test has been silent) without waiting for a hang.
+ [ $(( hb % 5 )) -eq 0 ] && echo "HANG WATCHDOG: alive; last test output ${silent}s ago"
+ if [ "$silent" -ge "$silent_limit" ] && [ "$dumps" -lt 3 ]; then
+ dumps=$(( dumps + 1 ))
+ pids="$(fork_pids | sort -un)"
+ # Safety net: if the fork JVM cannot be pinpointed, dump EVERY JVM.
+ [ -n "$pids" ] || pids="$(all_java_pids | sort -un)"
+ echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $pids))"
+ [ -n "$pids" ] || echo "HANG WATCHDOG: no java process found to dump"
+ for pid in $pids; do
+ # SIGQUIT makes the JVM print a full thread dump to its OWN stderr,
+ # which sbt relays into the test log via the SAME stream as test
+ # output -- so it lands in the job log even when a separately
+ # spawned jstack child's output would be buffered/lost. Also write
+ # jstack to a file for the per-shard artifact.
+ echo "----- SIGQUIT + jstack pid ${pid} -----"
+ kill -QUIT "$pid" 2>/dev/null || echo "HANG WATCHDOG: kill -QUIT failed for pid ${pid}"
+ timeout 120 "$JSTACK" -l "$pid" > "/tmp/threaddump-shard-${{ matrix.shard }}-${dumps}-${pid}.txt" 2>&1 \
+ || echo "HANG WATCHDOG: jstack failed/timed out for pid ${pid}"
+ done
+ echo "::endgroup::"
+ # The dump is now captured (job log via SIGQUIT + artifact via
+ # jstack file). A hung JVM otherwise stalls the whole shard until
+ # the 350-min job timeout AND keeps the job log frozen so the dump
+ # never becomes reachable. So KILL the wedged JVM(s): the suite
+ # fails fast (acceptable -- errors are expected; only an
+ # unrecoverable hang blocks CI), the job proceeds/ends, and the log
+ # + artifacts flush. Give SIGQUIT a moment to print first.
+ sleep 20
+ echo "HANG WATCHDOG: killing wedged JVM(s) to unblock the shard: $(printf '%s ' $pids)"
+ for pid in $pids; do kill -KILL "$pid" 2>/dev/null; done
+ fi
+ done
+ ) &
+ WATCHDOG_PID=$!
+
+ ./build/sbt \
+ -DsparkVersion=${{ steps.resolve.outputs.spark_version }} \
+ -v \
+ -J-XX:+UseG1GC -J-Xms512m -J-Xmx4G \
+ -J-XX:G1PeriodicGCInterval=10000 \
+ -J-XX:G1PeriodicGCSystemLoadThreshold=0 \
+ -J-XX:-G1PeriodicGCInvokesConcurrent \
+ -J-XX:MinHeapFreeRatio=5 -J-XX:MaxHeapFreeRatio=15 \
+ "++ ${DELTA_SCALA_VERSION}" \
+ 'set spark / Test / javaOptions ++= Seq("-Xmx2G", "-XX:+HeapDumpOnOutOfMemoryError", "-XX:HeapDumpPath=/tmp/")' \
+ 'set spark / Test / testOptions += Tests.Argument(TestFrameworks.ScalaTest, "-u", "target/test-reports")' \
+ "spark/test" 2>&1 | tee "$SBT_LOG"
+ SBT_EXIT=${PIPESTATUS[0]}
+ touch /tmp/sbt-done
+ kill "$WATCHDOG_PID" 2>/dev/null || true
+ set -e
+ echo "sbt spark/test exited with ${SBT_EXIT}"
+
+ # Memory forensics: a sudden forked-JVM death with no hs_err and no heap
+ # dump is almost always a kernel/cgroup OOM-kill (Velox off-heap + JVM
+ # heap exceeding the ~16G runner). Surface the cgroup peak + oom_kill
+ # count so we can confirm/measure it (cgroup v2 paths; best-effort).
+ ( echo "=== cgroup memory forensics (exit ${SBT_EXIT}) ==="
+ for f in /sys/fs/cgroup/memory.peak /sys/fs/cgroup/memory.max \
+ /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.events; do
+ [ -r "$f" ] && { echo "--- $f ---"; cat "$f"; }
+ done ) || true
+
+ # A compile/launch failure leaves no reports at all. In that case the
+ # gate would see zero failures and pass spuriously, so fail loudly.
+ REPORT_COUNT=$(find . -path '*/target/test-reports/*.xml' 2>/dev/null | wc -l || true)
+ echo "Found ${REPORT_COUNT} JUnit XML report file(s)."
+ if [ "${REPORT_COUNT}" -eq 0 ]; then
+ echo "::error::sbt produced no test reports (exit ${SBT_EXIT}) -- likely a compile or launch failure, not test failures."
+ exit 1
+ fi
+
+ # update_baseline=true -> SEED mode (record failures, never fail) so the
+ # baseline can be (re)generated. Otherwise ENFORCE against the baseline.
+ GATE_MODE=enforce
+ if [ "${{ steps.resolve.outputs.update_baseline }}" = "true" ]; then
+ GATE_MODE=seed
+ fi
+ mkdir -p "$GITHUB_WORKSPACE/gate-out"
+ python3 "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/compare-test-results.py" \
+ --mode "${GATE_MODE}" \
+ --reports-dir "$GITHUB_WORKSPACE/delta" \
+ --known-failures "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/known-failures.txt" \
+ --flaky-tests "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/flaky-tests.txt" \
+ --failures-out "$GITHUB_WORKSPACE/gate-out/failures-shard-${{ matrix.shard }}.txt" \
+ --ran-out "$GITHUB_WORKSPACE/gate-out/ran-shard-${{ matrix.shard }}.txt" \
+ --fail-on-fixed "${{ steps.resolve.outputs.fail_on_fixed }}"
Review Comment:
Would it be possible to extract these commands, in whole or in part, into a script under .github/workflows/util? This would improve the readability of the workflow.
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -93,6 +102,23 @@ jobs:
name: velox-native-lib-centos-7-${{github.sha}}
path: ./cpp/build/
if-no-files-found: error
+ # Consumed by the reusable Delta workflow (delta-spark-ut job).
+ - uses: actions/upload-artifact@v4
+ with:
+ name: velox-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
Review Comment:
Uploading the Arrow JARs is no longer necessary, as the public Apache Arrow JARs have been compatible with Gluten since last month.
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -93,6 +102,23 @@ jobs:
name: velox-native-lib-centos-7-${{github.sha}}
path: ./cpp/build/
if-no-files-found: error
+ # Consumed by the reusable Delta workflow (delta-spark-ut job).
+ - uses: actions/upload-artifact@v4
+ with:
+ name: velox-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ # Delta Spark UT, run via the reusable workflow so it reuses the native lib +
+ # arrow jars built above instead of duplicating the native build. Not gated on
+ # Delta-only paths: core/velox/substrait/cpp/shims changes can affect Delta
+ # query offload, so this runs on every trigger like the other spark-test jobs.
+ delta-spark-ut:
+ needs: build-native-lib-centos-7
+ uses: ./.github/workflows/delta_spark_ut.yml
+ with:
+ native_lib_artifact: velox-native-lib-centos-7-${{ github.sha }}
+ arrow_jars_artifact: velox-arrow-jars-centos-7-${{ github.sha }}
Review Comment:
ditto.
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,692 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. `spark_version` is the single source of truth for the
+ # Spark version: it drives the Gluten bundle profile (-Pspark-<v>), the bundle
+ # jar name, and Delta's -DsparkVersion, so the tests always run against a bundle
+ # built for the same Spark version (no separate value to keep in sync). Scala
+ # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from
+ # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a
+ # compatible delta_ref.
+ GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version }}
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version }}
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
Review Comment:
Not a blocker, just wanted to confirm whether we really need to test all 4 shard numbers (we're optimizing the CI pipeline to reduce GitHub runner usage, per Apache's policy).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3669332015
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -101,6 +106,80 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR
+ # touches high-signal Delta paths -- the Delta integration code
+ # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own
+ # files -- or carries the `run-delta-ci` opt-in label. Changes to general
+ # Velox/core/native code can also affect Delta offload but are touched
+ # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml
+ # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage
+ # down. NOTE: the label is read from the event that triggered this run, so add
+ # it before/with a push; labeling an already-finished run needs a new push.
+ delta-changes:
+ runs-on: ubuntu-22.04
+ outputs:
+ run_delta: ${{ steps.filter.outputs.run_delta }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ - name: Detect Delta-relevant changes / opt-in label
+ id: filter
+ env:
+ HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
Review Comment:
Invalid, duplicated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3522306262
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,685 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Delta `-DsparkVersion` value (must match the Gluten -P profile below)'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. Spark 4.1 + Scala 2.13 + JDK 17 matches Delta v4.2.0's
+ # default Spark version (4.1.0) from project/CrossSparkVersions.scala.
+ GLUTEN_SPARK_PROFILE: 'spark-4.1'
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: '4.1'
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Every input has a default (workflow_call + workflow_dispatch), so they
+ # are always set; just surface them as step outputs for the steps below.
+ env:
+ DELTA_REF: ${{ inputs.delta_ref }}
+ SPARK_VERSION: ${{ inputs.spark_version }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline }}
+ FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
Review Comment:
`spark_version` is user-configurable (workflow_dispatch / workflow_call), but the workflow always builds the Gluten bundle with the hard-coded Spark 4.1 profile (`GLUTEN_SPARK_PROFILE`/`GLUTEN_BUNDLE_SPARK_VERSION`). A manual run with `spark_version != 4.1` will run Delta tests for that Spark version against a Spark-4.1-compiled bundle, which is very likely to fail in confusing ways. Add an explicit validation here (or remove the input and hard-code Spark 4.1) so misconfigured runs fail fast with a clear message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5181837958
@philo-he @zhouyuan gentle reminder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5027637160
> > For the maintainers @philo-he @zhztheplayer : this runs on every velox_backend_x86 trigger. Fair reasoning, but with #12288 going on it should be a deliberate choice — every PR, Delta paths only, or nightly + label. Any works for me.
>
> @felipepessoto, we should avoid unnecessary GHA resource consumption, since our usage is still running high. That sometimes means making tradeoffs, particularly for failure cases that are rare. With that in mind, I'd suggest scheduling a nightly job that runs the full test set against the latest main, and narrowing the per-PR Delta tests to the most relevant cases that guard the key functionality. The per-PR job could also be triggered only by changes to the Delta module and a few core-related modules, so PRs that don't touch those paths skip it entirely. This keeps fast, focused signal on every PR while still catching the rarer regressions once a day.
Thanks @philo-he, that makes sense - I've implemented it in a82ab8945.
**Nightly full run.** `delta_spark_ut.yml` now has a `schedule` trigger (05:00 UTC) that runs the full suite against the latest `main`. It builds its own native lib (no caller to provide one) and runs with `fail_on_fixed=true`, so if a baselined test starts passing the nightly goes red, our signal to refresh `known-failures.txt`.
**Narrowed per-PR trigger.** A small `delta-changes` job now gates the Delta suite in `velox_backend_x86.yml`: it runs only when the PR touches a high-signal Delta path, or carries a `run-delta-ci` opt-in label (for PRs that change general code but still want to check Delta).
To pick where to draw the per-PR line, I grouped the candidate paths:
- **Group A - Delta-specific** (rare, ~11% of commits): `gluten-delta/**`, `delta_spark_ut.yml`, `util/delta-spark-ut/**`, `velox_backend_x86.yml`.
- **Group B - core offload code** (touched on ~60% of commits): `backends-velox/**`, `gluten-core/**`, `gluten-substrait/**`, `gluten-arrow/**`, `cpp/**`, `shims/**`.
- **Group C - build/infra** (broad, indirect): `dev/**`, `ep/build-velox/**`, `package/**`, `pom.xml`, `build/mvn`.
Keeping B+C would have triggered the suite on ~60% of PRs, barely a saving. So, I went with the **aggressive** option: **Group A plus the Delta integration code `backends-velox/src-delta*`** (the `src-delta`, `src-delta33`, `src-delta40` sources), and dropped B and C. That takes the per-PR trigger rate to ~17% of recent commits. General Velox/core/native changes that can affect Delta offload are covered by the nightly run and the `run-delta-ci` label rather than running on every PR.
One thing to flag: if you'd like the Delta checks to be required for merge, note they won't post on PRs where the suite is intentionally skipped, so they'd need to be non-required, or backed by an always-green shim job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3487629902
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,467 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline is empty (not yet bootstrapped) the mode automatically
+ degrades to ``seed`` so the first run is never spuriously red.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard).
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import glob
+import os
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped) sets of (suite, test) tuples. A test is
+ 'failed' if its <testcase> has a <failure> or <error> child, 'skipped' if
+ it has a <skipped> child, otherwise 'passed'. Suite-level aborts (a
+ <testsuite> reporting errors/failures with no failing <testcase>) are
+ recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = (suite, name)
+ tags = _child_local_tags(tc)
+ if "failure" in tags or "error" in tags:
+ failed.add(key)
+ suite_has_failing_tc = True
+ elif "skipped" in tags:
+ skipped.add(key)
+ else:
+ passed.add(key)
+
+ # Suite-level abort: counters say something failed but no testcase
+ # carried the failure (the suite blew up in beforeAll/constructor).
+ # Record a
+ # synthetic entry so the regression is visible.
+ try:
+ errors = int(ts.get("errors", "0") or "0")
+ failures = int(ts.get("failures", "0") or "0")
+ except ValueError:
+ errors = failures = 0
+ if (errors + failures) > 0 and not suite_has_failing_tc:
+ failed.add((suite_name, SUITE_ABORTED))
+
+ if not parsed_any:
+ eprint(
+ "WARNING: no JUnit <testsuite> elements found under {}".format(reports_dir)
+ )
+
+ # A test can't be both passed and failed; failure wins. Skipped only counts
+ # if the test was not otherwise seen (e.g. retried).
+ passed -= failed
+ skipped -= failed
+ skipped -= passed
+ return passed, failed, skipped
+
+
+# --------------------------------------------------------------------------- #
+# Reporting helpers
+# --------------------------------------------------------------------------- #
+def _summary_sink():
+ """Return a writer that mirrors to GITHUB_STEP_SUMMARY when available."""
+ path = os.environ.get("GITHUB_STEP_SUMMARY")
+ handle = open(path, "a", encoding="utf-8") if path else None
+
+ def write(line=""):
+ print(line)
+ if handle:
+ handle.write(line + "\n")
+
+ return write, handle
+
+
+def _print_block(write, title, entries, limit=50):
+ write("")
+ write("### {} ({})".format(title, len(entries)))
+ if not entries:
+ return
+ write("")
+ write("```")
+ for i, (suite, test) in enumerate(sorted(entries)):
+ if i >= limit:
+ write("... and {} more".format(len(entries) - limit))
+ break
+ write(format_entry(suite, test))
+ write("```")
+
+
+# --------------------------------------------------------------------------- #
+# Modes
+# --------------------------------------------------------------------------- #
+def run_enforce(args):
+ baseline = load_entries(args.known_failures)
+ passed, failed, skipped = parse_reports(args.reports_dir)
+
+ # Always emit this shard's artifacts for the aggregation job.
+ if args.failures_out:
+ write_entries(args.failures_out, failed)
+ if args.ran_out:
+ write_entries(args.ran_out, passed | failed)
Review Comment:
`--ran-out` is used in aggregate mode to detect stale baseline entries (`baseline - union_ran`). Excluding skipped tests from `ran_out` can incorrectly mark baseline entries as stale even though the test is still present and reported (but skipped). Consider including skipped tests in `ran_out` so stale detection reflects whether a test was seen in the reports, regardless of status.
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,672 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Delta `-DsparkVersion` value (must match the Gluten -P profile below)'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. Spark 4.1 + Scala 2.13 + JDK 17 matches Delta v4.2.0's
+ # default Spark version (4.1.0) from project/CrossSparkVersions.scala.
+ GLUTEN_SPARK_PROFILE: 'spark-4.1'
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: '4.1'
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix.
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar | head -n 1)
Review Comment:
With `set -euo pipefail`, this `ls ... | head` command will exit the step immediately when the glob matches nothing, so the explicit error handling below never runs and the logs become much less actionable. Make the glob non-fatal and let the subsequent check produce the intended error message.
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,672 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Delta `-DsparkVersion` value (must match the Gluten -P profile below)'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. Spark 4.1 + Scala 2.13 + JDK 17 matches Delta v4.2.0's
+ # default Spark version (4.1.0) from project/CrossSparkVersions.scala.
+ GLUTEN_SPARK_PROFILE: 'spark-4.1'
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: '4.1'
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix.
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar | head -n 1)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Every input has a default (workflow_call + workflow_dispatch), so they
+ # are always set; just surface them as step outputs for the steps below.
+ env:
+ DELTA_REF: ${{ inputs.delta_ref }}
+ SPARK_VERSION: ${{ inputs.spark_version }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline }}
+ FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
+
+ - name: Download Gluten bundle jar
+ uses: actions/download-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: gluten-bundle
+
+ - name: Install minimal tools
+ run: |
+ set -euo pipefail
+ # apache/gluten:centos-9-jdk17 already has java-17, git, tar, a POSIX
+ # shell, and curl-minimal (which provides the `curl` command sbt's
+ # launcher needs). Install the rest of what Delta's build/sbt and the
+ # tests may need. We deliberately do NOT install the full `curl`
+ # package -- it conflicts with the pre-installed curl-minimal.
+ yum install -y java-17-openjdk-devel which findutils gzip python3
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ git --version
+ curl --version | head -n 1
+
+ - name: Cache sbt / Ivy / Coursier
+ uses: actions/cache@v4
+ with:
+ path: |
+ /root/.sbt
+ /root/.ivy2
+ /root/.cache/coursier
+ # Intentionally NOT keyed by ${{ matrix.shard }} -- all shards share
+ # the same dependency tree, so a single shared cache (with parallel
+ # save races resolved by GH on a first-write-wins basis) gives the
+ # best storage / hit-rate tradeoff.
+ key: delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-${{ env.DELTA_SCALA_VERSION }}
+ restore-keys: |
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-
+
+ - name: Clone and patch Delta
+ run: |
+ set -euo pipefail
+ GLUTEN_JAR=$(ls "$GITHUB_WORKSPACE"/gluten-bundle/gluten-velox-bundle-spark*_*-linux_amd64-*.jar | head -n 1)
+ echo "Using Gluten bundle: $GLUTEN_JAR"
Review Comment:
This `ls ... | head` will fail the step with a generic `ls: cannot access ...` error if the artifact download produced no matching jar (or the naming changes). Since the next line logs the chosen jar, it’s better to make the lookup non-fatal and emit a clear, actionable error if no jar is found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3487637705
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,206 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# Shallow clone the requested tag/branch. Fall back to full clone when the ref is a SHA.
+if ! git clone --depth 1 --branch "$DELTA_REF" https://github.com/delta-io/delta.git "$DELTA_DIR"; then
+ echo "Shallow clone of ref '${DELTA_REF}' failed, falling back to full clone."
+ rm -rf "$DELTA_DIR"
+ git clone https://github.com/delta-io/delta.git "$DELTA_DIR"
+ git -C "$DELTA_DIR" checkout "$DELTA_REF"
+fi
+git -C "$DELTA_DIR" --no-pager log -1 --oneline
+echo "::endgroup::"
+
+echo "::group::Injecting Gluten bundle jar onto the spark project's TEST classpath"
+# The Gluten bundle jar must be on the spark project's TEST runtime classpath
+# (so DeltaSQLCommandTest can load org.apache.gluten.GlutenPlugin by name) but
+# NOT on the COMPILE classpath of `sparkV1`, which is the project that holds
+# Delta's main sources. The bundle's transitive contents include extra symbols
+# under `org.apache.spark.sql` that collide with Delta's main sources -- e.g.
+# MergeOutputGeneration.scala imports both `org.apache.spark.sql._` and
+# `org.apache.spark.sql.delta.ClassicColumnConversions._`, and would then fail
+# with `reference to expression is ambiguous`.
+#
+# sbt auto-scans `<baseDirectory>/lib` via `unmanagedBase`. Two relevant
+# projects in Delta v4.2.0 have a `lib/` baseDirectory:
+# - sparkV1: `project in file("spark")` -> spark/lib
+# - spark : `project in file("spark-unified")` -> spark-unified/lib
+# unmanagedJars are project-scoped (NOT inherited by dependents), so dropping
+# the bundle into spark-unified/lib/ adds it to the unified `spark` project's
+# Compile *and* Test classpaths -- but NOT to sparkV1's. That's exactly what
+# we want:
+# * sparkV1/Compile sees ONLY Delta's regular deps -> Delta main compiles.
+# * spark/Test/fullClasspath sees the bundle -> tests load GlutenPlugin.
+# (Verified empirically: with bundle only in spark-unified/lib/, sbt's
+# `show sparkV1/Compile/dependencyClasspath` excludes the bundle and
+# `show spark/Test/fullClasspath` includes it.)
+#
+# We deliberately do NOT also drop the bundle into spark/lib/, which is what
+# caused the previous compile failure: spark/lib/ is sparkV1's unmanagedBase,
+# and putting the bundle there would re-introduce the ambiguity errors.
+SPARK_UNIFIED_LIB="$DELTA_DIR/spark-unified/lib"
+mkdir -p "$SPARK_UNIFIED_LIB"
+cp "$GLUTEN_BUNDLE_JAR" "$SPARK_UNIFIED_LIB/gluten-velox-bundle.jar"
+ls -lh "$SPARK_UNIFIED_LIB"
+echo "::endgroup::"
+
+echo "::group::Patching DeltaSQLCommandTest to enable Gluten plugin"
+TARGET="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$TARGET" ]; then
+ echo "Expected file not found in Delta clone: $TARGET" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
+ exit 1
+fi
+cp "$PATCH_SOURCE" "$TARGET"
+echo "Patched $TARGET"
+echo "--- diff vs. upstream ---"
+git -C "$DELTA_DIR" --no-pager diff -- "spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala" || true
+echo "::endgroup::"
+
+# Delta's tests collect file-source scans by matching the concrete
+# `FileSourceScanExec` case class; Gluten offloads the scan to
+# DeltaScanTransformer, a `FileSourceScanLike` sibling, so those matches miss
+# (`scala.MatchError: List()`, empty partition filters, broken column-pruning /
+# scan-metric checks across many suites). delta-io/delta#7104 and #7105 widen the
+# matches to the shared `FileSourceScanLike` interface that both the vanilla and
+# Gluten scans implement (behavior-preserving for vanilla). Both are merged
+# upstream but land after the pinned DELTA_REF (v4.2.0), so apply them here; once
+# DELTA_REF includes a commit its cherry-pick is a clean no-op and the call can go.
+#
+# Depth-2 fetch brings each fix commit and its parent, which cherry-pick needs to
+# diff against (a depth-1 fetch grafts the parent away); `-n` stages the change
+# without requiring a committer identity.
+cherry_pick_delta_fix() {
+ local sha="$1" pr="$2"
+ echo "Cherry-picking delta-io/delta${pr}"
+ git -C "$DELTA_DIR" fetch --quiet --depth 2 origin "$sha"
+ git -C "$DELTA_DIR" cherry-pick -n "$sha"
+}
Review Comment:
A few clarifications on the mechanics, then where I've landed.
The script uses git cherry-pick -n ( --no-commit ), not a committing cherry-pick. The failure you describe - an "empty cherry-pick" leaving the repo in CHERRY_PICK_HEAD - is specific to the committing path: that's where git stops with "The previous cherry-pick is now empty" and waits for --skip / --continue . With -n , re-applying a commit that's already present is a clean no-op: it applies the (empty) diff to the index, exits 0, and never creates CHERRY_PICK_HEAD . I verified this directly. The only way -n errors on an already-present commit is a genuine content conflict (e.g. neighbouring lines changed), which surfaces loudly as a non-zero exit with conflict markers - not a silent half-finished cherry-pick.
For the pinned ref this is moot anyway: delta_ref is v4.2.0 , which predates both fixes (#7104, #7105), so the cherry-picks apply cleanly and the suite is green. The calls are meant to be deleted when delta_ref is bumped to a release that already contains the fixes - the comment block above them says exactly that - so we don't rely on re-application being a no-op in steady state.
On the suggested guard: I'd avoid merge-base --is-ancestor specifically, because the Delta checkout is a shallow --depth 1 clone, so ancestry can't be resolved across the shallow boundary - the check would give false negatives and still attempt the cherry-pick.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3487628565
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,467 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline is empty (not yet bootstrapped) the mode automatically
+ degrades to ``seed`` so the first run is never spuriously red.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard).
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import glob
+import os
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped) sets of (suite, test) tuples. A test is
+ 'failed' if its <testcase> has a <failure> or <error> child, 'skipped' if
+ it has a <skipped> child, otherwise 'passed'. Suite-level aborts (a
+ <testsuite> reporting errors/failures with no failing <testcase>) are
+ recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = (suite, name)
+ tags = _child_local_tags(tc)
+ if "failure" in tags or "error" in tags:
+ failed.add(key)
+ suite_has_failing_tc = True
+ elif "skipped" in tags:
+ skipped.add(key)
+ else:
+ passed.add(key)
+
+ # Suite-level abort: counters say something failed but no testcase
+ # carried the failure (the suite blew up in beforeAll/constructor).
+ # Record a
+ # synthetic entry so the regression is visible.
+ try:
+ errors = int(ts.get("errors", "0") or "0")
+ failures = int(ts.get("failures", "0") or "0")
+ except ValueError:
+ errors = failures = 0
+ if (errors + failures) > 0 and not suite_has_failing_tc:
+ failed.add((suite_name, SUITE_ABORTED))
+
+ if not parsed_any:
+ eprint(
+ "WARNING: no JUnit <testsuite> elements found under {}".format(reports_dir)
+ )
+
+ # A test can't be both passed and failed; failure wins. Skipped only counts
+ # if the test was not otherwise seen (e.g. retried).
+ passed -= failed
+ skipped -= failed
+ skipped -= passed
+ return passed, failed, skipped
+
+
+# --------------------------------------------------------------------------- #
+# Reporting helpers
+# --------------------------------------------------------------------------- #
+def _summary_sink():
+ """Return a writer that mirrors to GITHUB_STEP_SUMMARY when available."""
+ path = os.environ.get("GITHUB_STEP_SUMMARY")
+ handle = open(path, "a", encoding="utf-8") if path else None
+
+ def write(line=""):
+ print(line)
+ if handle:
+ handle.write(line + "\n")
+
+ return write, handle
+
+
+def _print_block(write, title, entries, limit=50):
+ write("")
+ write("### {} ({})".format(title, len(entries)))
+ if not entries:
+ return
+ write("")
+ write("```")
+ for i, (suite, test) in enumerate(sorted(entries)):
+ if i >= limit:
+ write("... and {} more".format(len(entries) - limit))
+ break
+ write(format_entry(suite, test))
+ write("```")
+
+
+# --------------------------------------------------------------------------- #
+# Modes
+# --------------------------------------------------------------------------- #
+def run_enforce(args):
+ baseline = load_entries(args.known_failures)
+ passed, failed, skipped = parse_reports(args.reports_dir)
+
+ # Always emit this shard's artifacts for the aggregation job.
+ if args.failures_out:
+ write_entries(args.failures_out, failed)
+ if args.ran_out:
+ write_entries(args.ran_out, passed | failed)
Review Comment:
Thanks - you're right that ran-out excludes skipped tests, so stale = baseline - union_ran would count a skipped baseline entry as "stale". In this pipeline that can't actually happen in a way that matters, so I'm leaving it as-is for now:
• A skipped test never enters the baseline. The aggregate writes the refreshed baseline as union_failed only, so a skipped (non-failing) test is never recorded. For the mislabel to occur, an entry would have to flip from failed (when the baseline was captured) to skipped (in a later run).
• The ref is pinned, so that flip doesn't occur. delta_ref is pinned to v4.2.0 and the test environment is fixed (cloud filesystems disabled, etc.), so the passed/failed/skipped partition is deterministic - a test is consistently failed or consistently skipped. For the same reason suites are never renamed or removed, so stale is normally empty.
• stale is informational only. It doesn't affect the exit code (only regressions gates the aggregate job), and the per-shard enforce gate ignores skipped entirely ( regressions = failed - baseline , fixed = baseline & passed ). So even a mislabel can't fail CI or corrupt the baseline - a skipped test simply drops out of the regenerated union_failed , which is correct.
If we start bumping delta_ref regularly, emitting a per-shard skipped list and using stale = baseline - union_ran - union_skipped is the right fix (and, as you note, folding skipped into ran-out instead would wrongly count skipped as now-passing). I'll revisit it then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3661100391
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,820 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline file exists but is empty (not yet bootstrapped) the mode
+ automatically degrades to ``seed`` so the first run is never spuriously red.
+ A *missing* ``--known-failures`` file is treated as a configuration error
+ (the gate fails) so a mis-referenced path can't silently pass.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` / ``--skipped-out``
+ file into a single, sorted, ready-to-commit ``known-failures.txt`` and report
+ stale baseline entries (tests no longer present in any shard). Skipped tests
+ are tracked apart from run tests so "stale" means *truly absent* rather than
+ merely skipped this run; they are also kept out of the now-passing set, since
+ a skipped test is not evidence of a fix. Pass ``--expected-shards N``
+ to fail when fewer than ``N`` shards contributed gate lists (a shard that
+ died before writing them), so an incomplete baseline is never produced.
+
+Flaky quarantine (``--flaky-tests``)
+ Some Delta-on-Gluten failures are non-deterministic (e.g. a native bug that
+ only triggers on certain runtime plans), so they are neither a stable pass
+ nor a stable failure and cannot live in the baseline: baselining them turns
+ the gate red on every run where they pass, and leaving them out turns it red
+ on every run where they fail. ``flaky-tests.txt`` quarantines them -- a
+ quarantined test never counts as a regression (when it fails) nor as
+ now-passing (when it passes), and is excluded from the regenerated baseline.
+ Its SUITE is an fnmatch glob (so one line covers a root-cause family across
+ generated suite variants); its TEST name is matched exactly.
+
+Flaky quarantine by error signature (``--flaky-error-patterns``)
+ When a failure is caused by a known nondeterministic bug that surfaces on a
+ *different test each run* (e.g. the native Delta DV bitmap row-index error),
+ matching by test name is whack-a-mole. ``flaky-error-patterns.txt`` instead
+ quarantines by root cause: each line is a regex matched against a failed
+ test's <failure>/<error> text, and any failure that matches is treated as
+ flaky regardless of which test it landed on.
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import fnmatch
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+
+class NoReportsError(RuntimeError):
+ """Raised when no JUnit <testsuite> elements are found under reports_dir."""
+
+
+class CorruptReportError(NoReportsError):
+ """Raised when an expected JUnit report file (TEST-*.xml) fails to parse.
+
+ Subclasses NoReportsError so the enforce/seed handler treats a truncated
+ report as a hard data error (exit 2) instead of silently dropping the
+ suite's results and letting the gate pass on partial data.
+ """
+
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def normalize_key(suite, test):
+ """Normalize a (suite, test) key parsed from JUnit XML to match baseline keys.
+
+ Baseline/flaky entries round-trip through write_entries (which collapses CR/LF
+ in the test name to spaces) and parse_entry (which strips the whole
+ ``suite#test`` line). A raw XML name carrying a trailing newline or
+ surrounding whitespace would therefore never equal its normalized baseline
+ entry: the gate would keep reporting it as a REGRESSION, and the
+ copy-pasteable line it prints could never suppress it (load strips it back).
+ Delta test names are freeform, so a version bump could introduce exactly that.
+ Applying the identical format+parse round-trip here keeps the two sides in
+ sync (and is a no-op for the normal, whitespace-free names).
+ """
+ safe_test = (test or "").replace("\r", " ").replace("\n", " ")
+ normalized = parse_entry(format_entry(suite or "", safe_test))
+ # parse_entry only returns None for a blank/comment line, which a real
+ # testcase key is not; fall back to a bare strip to keep this total.
+ if normalized is None:
+ return ((suite or "").strip(), safe_test.strip())
+ return normalized
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def make_is_flaky(flaky_entries):
+ """Build a predicate that matches a (suite, test) tuple against flaky entries.
+
+ A flaky entry quarantines a test whose failure is known to be non-deterministic
+ (see flaky-tests.txt). The entry's SUITE is treated as an fnmatch glob so a
+ single line can cover a root-cause family across generated suite variants
+ (e.g. ``*DVs*Suite`` matches every deletion-vector merge suite, ``*`` matches
+ any suite); the TEST name is matched exactly (test names are freeform and may
+ contain glob metacharacters, so they are never globbed).
+ """
+ exact = set()
+ globbed = []
+ for suite, test in flaky_entries:
+ if any(ch in suite for ch in "*?["):
+ globbed.append((suite, test))
+ else:
+ exact.add((suite, test))
+
+ def is_flaky(entry):
+ if entry in exact:
+ return True
+ suite, test = entry
+ for glob_suite, glob_test in globbed:
+ if test == glob_test and fnmatch.fnmatchcase(suite, glob_suite):
+ return True
+ return False
+
+ return is_flaky
+
+
+def load_patterns(path):
+ """Load flaky-error regex patterns from a file (one per line).
+
+ Blank lines and ``#`` comments are ignored. Each remaining line is compiled
+ as a case-sensitive regex. These match the FAILURE TEXT of a failed test
+ (its JUnit <failure>/<error> message + stack), so a test that fails with a
+ known-nondeterministic native error (e.g. the Delta DV bitmap row-index bug)
+ can be quarantined by root cause instead of by exact test name.
+ """
+ patterns = []
+ if not path or not os.path.exists(path):
+ return patterns
+ with open(path, encoding="utf-8") as fh:
+ for line in fh:
+ line = line.rstrip("\n")
+ if not line.strip() or line.lstrip().startswith("#"):
+ continue
+ patterns.append(re.compile(line))
+ return patterns
Review Comment:
Valid — fixed in 9a8772137. `flaky-error-patterns.txt` is hand-edited, so a typo is plausible, and the bare `re.error` named neither the file nor the line.
`load_patterns()` now tracks the line number and raises a `BadPatternError` that the gate catches and reports before exiting 2:
```
ERROR: .../flaky-error-patterns.txt:4: invalid regex '[unclosed': unterminated character set at position 0
```
Verified: exit code 2, message carries file:line and the offending pattern, and no traceback leaks.
##########
.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt:
##########
@@ -0,0 +1,43 @@
+# Flaky-error signatures for the Delta Spark UT (Gluten) gate.
+#
+# Each non-comment line is a Python regex matched (re.search, case-sensitive)
+# against a failed test's JUnit <failure>/<error> text (message + stack). A
+# failure that matches is QUARANTINED by root cause: it never counts as a
+# regression, and is dropped from the shard's failures list so it can't leak
+# into the regenerated baseline -- regardless of WHICH test it landed on.
+#
+# Use this (instead of flaky-tests.txt) when a known nondeterministic bug
+# surfaces on a different test each run, so matching by test name is
+# whack-a-mole. Prefer fixing the underlying bug and REMOVING the entry.
+#
+# ---------------------------------------------------------------------------
+# Native Delta bitmap aggregator receives an INVALID row index during a MERGE
+# that writes deletion vectors, and aborts. The garbage row index trips one of
+# two bounds checks, so the same root cause shows up with two messages:
+#
+# too large (Long.MAX_VALUE):
+# VeloxRuntimeError INVALID_STATE
+# Reason: Delta RoaringBitmapArray row index 9223372036854775807 exceeds max
+# representable value 9223372030412324864
+# Expression: value <= kMaxRepresentableValue
+# Function: addSafe File: .../velox/compute/delta/RoaringBitmapArray.cpp:92
+#
+# negative (garbage):
+# VeloxRuntimeError INVALID_STATE
+# Reason: Delta bitmap row index cannot be negative: -6254810385378525259
+# Expression: value >= 0
+# Function: addRowIndex File: .../operators/functions/delta/DeltaBitmapAggregator.cc:44
+#
+# Both are the same root cause (garbage row index into the DV bitmap aggregator)
+# but distinct native errors, so each has its own explicit pattern below --
+# deliberately specific (bound to the exact error string) rather than a broad
+# "Delta bitmap row index" match, to avoid masking an unrelated failure. Add a
+# new line if a further bounds-check variant appears. Intermittent (depends on
+# the runtime plan/scan/scheduling), so it hits a different *DVs*Suite MERGE test
+# on each run. Remove these once the row-index materialization is fixed in the
+# native backend. Tracked upstream (DV bitmap invalid row index).
+# ---------------------------------------------------------------------------
+# too-large (Long.MAX_VALUE) -- RoaringBitmapArray.cpp addSafe, value <= kMaxRepresentableValue
+Delta RoaringBitmapArray row index \d+ exceeds max representable value
+# negative garbage -- DeltaBitmapAggregator.cc addRowIndex, value >= 0
+Delta bitmap row index cannot be negative: -?\d+
Review Comment:
Agreed — tightened to `-\d+` in 9a8772137. The native check is `value >= 0`, so the index in that message is always negative, and this file's own guidance is to bind patterns to the exact error string "to avoid masking an unrelated failure" — `-?` quietly worked against that.
Verified against the real patterns file: the negative message is still quarantined, the hypothetical positive one no longer is, and the other signature is unaffected.
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,238 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# init + shallow fetch resolves a tag, branch OR commit SHA in a single path
+# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the
+# destructive `rm -rf "$DELTA_DIR"` it required. `--` terminates options so a
+# DELTA_REF starting with `-` can't be misread as a git flag (this script is
+# workflow_dispatch-runnable with a user-supplied ref).
+#
+# Every step here is idempotent so a local re-run (or a CI re-run on a runner
+# that kept the workspace) resumes instead of dying: `git init` re-initializes
+# an existing repo harmlessly, but `remote add` errors out when `origin` already
+# exists, so drop it first; and `checkout -f` discards leftovers from a previous
+# partial run. Nothing worth keeping exists here yet -- the bundle jar and the
+# source patches below are applied *after* this block.
+git init -q "$DELTA_DIR"
+git -C "$DELTA_DIR" remote remove origin 2>/dev/null || true
+git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git
+git -C "$DELTA_DIR" fetch -q --depth 1 origin -- "$DELTA_REF"
+git -C "$DELTA_DIR" checkout -qf FETCH_HEAD
+git -C "$DELTA_DIR" --no-pager log -1 --oneline
+echo "::endgroup::"
+
+echo "::group::Injecting Gluten bundle jar onto the spark project's TEST classpath"
+# The Gluten bundle jar must be on the spark project's TEST runtime classpath
+# (so DeltaSQLCommandTest can load org.apache.gluten.GlutenPlugin by name) but
+# NOT on the COMPILE classpath of `sparkV1`, which is the project that holds
+# Delta's main sources. The bundle's transitive contents include extra symbols
+# under `org.apache.spark.sql` that collide with Delta's main sources -- e.g.
+# MergeOutputGeneration.scala imports both `org.apache.spark.sql._` and
+# `org.apache.spark.sql.delta.ClassicColumnConversions._`, and would then fail
+# with `reference to expression is ambiguous`.
+#
+# sbt auto-scans `<baseDirectory>/lib` via `unmanagedBase`. Two relevant
+# projects in Delta v4.2.0 have a `lib/` baseDirectory:
+# - sparkV1: `project in file("spark")` -> spark/lib
+# - spark : `project in file("spark-unified")` -> spark-unified/lib
+# unmanagedJars are project-scoped (NOT inherited by dependents), so dropping
+# the bundle into spark-unified/lib/ adds it to the unified `spark` project's
+# Compile *and* Test classpaths -- but NOT to sparkV1's. That's exactly what
+# we want:
+# * sparkV1/Compile sees ONLY Delta's regular deps -> Delta main compiles.
+# * spark/Test/fullClasspath sees the bundle -> tests load GlutenPlugin.
+# (Verified empirically: with bundle only in spark-unified/lib/, sbt's
+# `show sparkV1/Compile/dependencyClasspath` excludes the bundle and
+# `show spark/Test/fullClasspath` includes it.)
+#
+# We deliberately do NOT also drop the bundle into spark/lib/, which is what
+# caused the previous compile failure: spark/lib/ is sparkV1's unmanagedBase,
+# and putting the bundle there would re-introduce the ambiguity errors.
+SPARK_UNIFIED_LIB="$DELTA_DIR/spark-unified/lib"
+mkdir -p "$SPARK_UNIFIED_LIB"
+cp "$GLUTEN_BUNDLE_JAR" "$SPARK_UNIFIED_LIB/gluten-velox-bundle.jar"
+ls -lh "$SPARK_UNIFIED_LIB"
+echo "::endgroup::"
+
+echo "::group::Patching DeltaSQLCommandTest to enable Gluten plugin"
+TARGET="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$TARGET" ]; then
+ echo "Expected file not found in Delta clone: $TARGET" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
+ exit 1
+fi
+cp "$PATCH_SOURCE" "$TARGET"
+echo "Patched $TARGET"
+echo "--- diff vs. upstream ---"
+git -C "$DELTA_DIR" --no-pager diff -- "spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala" || true
+echo "::endgroup::"
+
+# Delta's tests collect file-source scans by matching the concrete
+# `FileSourceScanExec` case class; Gluten offloads the scan to
+# DeltaScanTransformer, a `FileSourceScanLike` sibling, so those matches miss
+# (`scala.MatchError: List()`, empty partition filters, broken column-pruning /
+# scan-metric checks across many suites). delta-io/delta#7104 and #7105 widen the
+# matches to the shared `FileSourceScanLike` interface that both the vanilla and
+# Gluten scans implement (behavior-preserving for vanilla). Both are merged
+# upstream but land after the pinned DELTA_REF (v4.2.0), so apply them here; once
+# DELTA_REF includes a fix, cherry_pick_delta_fix detects it and skips (see below).
+#
+# Depth-2 fetch brings each fix commit and its parent, which cherry-pick needs to
+# diff against (a depth-1 fetch grafts the parent away); `-n` stages the change
+# without requiring a committer identity.
+cherry_pick_delta_fix() {
+ local sha="$1" pr="$2"
+ git -C "$DELTA_DIR" fetch --quiet --depth 2 origin "$sha"
+ echo "Cherry-picking delta-io/delta${pr}"
+ if git -C "$DELTA_DIR" cherry-pick -n "$sha"; then
+ return 0
+ fi
+ # The cherry-pick did not apply. The usual cause is that the pinned DELTA_REF
+ # already contains this fix (e.g. after a version bump), which makes the patch
+ # empty/conflicting and would -- under `set -e` -- abort the whole setup. We
+ # can't use ancestry to tell "already contained" from a genuine conflict here
+ # (the clone is shallow, so `merge-base --is-ancestor` can't see past the graft),
+ # so recover the exact paths this fix touches -- leaving other setup such as the
+ # DeltaSQLCommandTest patch intact -- and continue. This is self-correcting: if
+ # the fix is genuinely still needed, the FileSourceScanLike failures it prevents
+ # resurface as gate regressions rather than being hidden by a hard abort here.
+ echo "Cherry-pick of delta-io/delta${pr} did not apply cleanly" \
+ "(most likely already contained in ${DELTA_REF}); skipping it."
+ local f
+ while IFS= read -r f; do
+ [ -n "$f" ] || continue
+ git -C "$DELTA_DIR" reset -q -- "$f" 2>/dev/null || true
+ git -C "$DELTA_DIR" checkout -q -- "$f" 2>/dev/null || true
+ done < <(git -C "$DELTA_DIR" diff-tree --no-commit-id --name-only -r "$sha")
+ # Clear any leftover sequencer state (harmless if none exists).
+ git -C "$DELTA_DIR" cherry-pick --quit 2>/dev/null || true
+ return 0
+}
+
+echo "::group::Cherry-picking upstream Delta FileSourceScanLike test fixes"
+cherry_pick_delta_fix 46bd45d57eadd7e528002a0ae7bd36ce5a456eca "#7104 (ScanReportHelper.collectScans)"
+cherry_pick_delta_fix 959e00e15f41f56afc1c9bb95d160c55c6dc7068 "#7105 (9 more test suites)"
+echo "::endgroup::"
+
+echo "::group::Force-failing memory-hog DeletionVectorsSuite 2B-row tests"
+# Two DeletionVectorsSuite tests read from / delete from a 2-billion-row table.
+# Under the Gluten Velox bundle they balloon the forked test JVM to ~13G of
+# NATIVE memory (row-index materialization) and the kernel/cgroup OOM-kills it.
+# The dead fork then wedges sbt, hanging the whole shard until the workflow's
+# hang-watchdog dumps threads and kills it (~16 min wasted, and every suite
+# QUEUED AFTER it in that fork is skipped) -- see delta_spark_ut.yml.
+#
+# Rather than silently `ignore` these (easy to forget), we make them FAIL FAST
+# with a clear message: the gap stays visible in the test reports / baseline
+# until the native memory blow-up is fixed, at which point this patch should be
+# removed. NOTE: making the suite complete also un-skips the rest of the shard's
+# suite queue, so the known-failures baseline must be refreshed after this.
+#
+# ORDER MATTERS: keep this sed AFTER the cherry-picks above. #7105 also edits
+# DeletionVectorsSuite.scala, and git cherry-pick aborts (exit 128) when the work
+# tree has uncommitted edits to a file it touches.
+DVS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala"
+if [ ! -f "$DVS" ]; then
+ echo "Expected file not found in Delta clone: $DVS" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed." >&2
+ exit 1
+fi
+# Inject `fail(...)` as the first statement of each test body (the line ending
+# in `) {`). Delta sets no -Xfatal-warnings / dead-code warning, so the now-
+# unreachable original body compiles fine. Keep each injected line <100 chars:
+# Delta's scalastyle enforces a 100-char line length on test sources. The full
+# rationale lives in this comment, so the in-test message stays terse.
+sed -i 's#huge table: read from tables of 2B rows with existing DV of many zeros") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV read OOMs the test JVM; see setup-delta.sh")#' "$DVS"
+sed -i 's#number of rows from tables of 2B rows with DVs") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV delete OOMs the test JVM; see setup-delta.sh")#' "$DVS"
+INJECTED=$(grep -c "Gluten CI] Force-failed" "$DVS" || true)
+if [ "$INJECTED" -ne 2 ]; then
Review Comment:
The premise is correct, but this was already fixed in 85fdc62ed (pushed just before this review) — the clone step now force-checks-out `FETCH_HEAD`, which resets this tracked file to upstream on every run, so the sed always starts from a pristine file.
I verified both directions rather than assume it:
| clone step | run 1 | run 2 | run 3 |
|---|---|---|---|
| `checkout -q` (old) | INJECTED=2 | **INJECTED=4 → abort** | INJECTED=6 → abort |
| `checkout -qf` (current) | INJECTED=2 | INJECTED=2 | INJECTED=2 |
So exactly the failure you describe, and it no longer reproduces. I didn't add a second reset before the sed since it would be redundant, but you did surface that the `-f` is now load-bearing for a block ~120 lines away, which wasn't obvious — so 9a8772137 notes that dependency in the existing "ORDER MATTERS" comment to stop someone dropping the flag later.
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,459 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib artifact it already built, so the expensive
+ # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a
+ # called workflow runs as part of the caller run), so the jobs below download it
+ # by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+ # Nightly full run against the latest default branch. The per-PR entry point
+ # (velox_backend_x86.yml) now runs the Delta suite only when a PR touches
+ # Delta-relevant paths (or carries the opt-in label), to save GHA minutes; this
+ # scheduled run keeps full coverage once a day so rarer regressions are still
+ # caught. It builds its own native lib (build-native-lib-centos-7 below) since
+ # there is no caller to provide one, and uses the workflow's default inputs.
+ schedule:
+ - cron: '0 5 * * *'
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
Review Comment:
Same answer as the two earlier threads on this line: the env var isn't introduced by this change — it predates it and is set in ~10 workflows across the repo, because several pinned actions still run on Node 20. Removing it only here would be inconsistent and would surface hard failures / deprecation noise for those actions. Migrating actions off Node 20 is a repo-wide change better handled separately.
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -101,6 +106,70 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR
+ # touches high-signal Delta paths -- the Delta integration code
+ # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own
+ # files -- or carries the `run-delta-ci` opt-in label. Changes to general
+ # Velox/core/native code can also affect Delta offload but are touched
+ # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml
+ # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage
+ # down. NOTE: the label is read from the event that triggered this run, so add
+ # it before/with a push; labeling an already-finished run needs a new push.
+ delta-changes:
+ runs-on: ubuntu-22.04
+ outputs:
+ run_delta: ${{ steps.filter.outputs.run_delta }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ - name: Detect Delta-relevant changes / opt-in label
+ id: filter
+ env:
+ HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
Review Comment:
Same answer as the earlier thread on this block: this workflow is triggered `on: pull_request` only, so `github.event.pull_request` is always present. Beyond that, GitHub Actions returns an empty string when you read a property of a missing context rather than failing evaluation, so there is no evaluation-time failure to guard against — the shell-level fail-open is just belt-and-suspenders. An `event_name == 'pull_request'` guard would be unconditionally true here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3661179827
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,459 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib artifact it already built, so the expensive
+ # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a
+ # called workflow runs as part of the caller run), so the jobs below download it
+ # by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+ # Nightly full run against the latest default branch. The per-PR entry point
+ # (velox_backend_x86.yml) now runs the Delta suite only when a PR touches
+ # Delta-relevant paths (or carries the opt-in label), to save GHA minutes; this
+ # scheduled run keeps full coverage once a day so rarer regressions are still
+ # caught. It builds its own native lib (build-native-lib-centos-7 below) since
+ # there is no caller to provide one, and uses the workflow's default inputs.
+ schedule:
+ - cron: '0 5 * * *'
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. `spark_version` is the single source of truth for the
+ # Spark version: it drives the Gluten bundle profile (-Pspark-<v>), the bundle
+ # jar name, and Delta's -DsparkVersion, so the tests always run against a bundle
+ # built for the same Spark version (no separate value to keep in sync). Scala
+ # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from
+ # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a
+ # compatible delta_ref.
+ GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version || '4.1' }}
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version || '4.1' }}
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone runs (workflow_dispatch + nightly schedule) build the native lib
+ # here. When called by velox_backend_x86.yml the caller already built it and
+ # passes it as an input, so this job is skipped and the duplicate build avoided.
+ if: github.event_name == 'workflow_dispatch' || github.event_name == 'schedule'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Surface the inputs as step outputs. workflow_call / workflow_dispatch
+ # supply them (with defaults); the nightly `schedule` event supplies NONE,
+ # so fall back to the same defaults here. The boolean inputs are rendered
+ # as explicit 'true'/'false' strings (never empty) via the `&&/||` form so
+ # a schedule run resolves cleanly: update_baseline=false (enforce) and
+ # fail_on_fixed=true (so a now-passing baseline test turns the nightly red
+ # -- our signal that the committed baseline needs refreshing).
+ env:
+ DELTA_REF: ${{ inputs.delta_ref || 'v4.2.0' }}
+ SPARK_VERSION: ${{ inputs.spark_version || '4.1' }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism || '4' }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline && 'true' || 'false' }}
+ FAIL_ON_FIXED: ${{ github.event_name == 'schedule' && 'true' || (inputs.fail_on_fixed && 'true' || 'false') }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
+
+ - name: Download Gluten bundle jar
+ uses: actions/download-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: gluten-bundle
+
+ - name: Install minimal tools
+ run: |
+ set -euo pipefail
+ # apache/gluten:centos-9-jdk17 already has java-17, git, tar, a POSIX
+ # shell, and curl-minimal (which provides the `curl` command sbt's
+ # launcher needs). Install the rest of what Delta's build/sbt and the
+ # tests may need. We deliberately do NOT install the full `curl`
+ # package -- it conflicts with the pre-installed curl-minimal.
+ yum install -y java-17-openjdk-devel which findutils gzip python3
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ git --version
+ curl --version | head -n 1
+
+ - name: Cache sbt / Ivy / Coursier
+ uses: actions/cache@v4
+ with:
+ path: |
+ /root/.sbt
+ /root/.ivy2
+ /root/.cache/coursier
+ # Intentionally NOT keyed by ${{ matrix.shard }} -- all shards share
+ # the same dependency tree, so a single shared cache (with parallel
+ # save races resolved by GH on a first-write-wins basis) gives the
+ # best storage / hit-rate tradeoff.
+ key: delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-${{ env.DELTA_SCALA_VERSION }}
+ restore-keys: |
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-
Review Comment:
I don't think this one holds — forward slashes are valid in `actions/cache` keys, so no change here.
The validation is in `actions/toolkit`, [`packages/cache/src/cache.ts`](https://github.com/actions/toolkit/blob/main/packages/cache/src/cache.ts), and the only character it rejects is a comma:
```ts
function checkKey(key: string): void {
if (key.length > 512) {
throw new ValidationError(
`Key Validation Error: ${key} cannot be larger than 512 characters.`
)
}
const regex = /^[^,]*$/
if (!regex.test(key)) {
throw new ValidationError(
`Key Validation Error: ${key} cannot contain commas.`
)
}
}
```
That check runs over `[primaryKey, ...restoreKeys]`, so it covers the `restore-keys` too, and the [caching reference](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching) likewise documents only the 512-character limit. Empirically, using `${{ github.ref }}` (i.e. `refs/heads/<branch>`, slashes and all) in a cache key is a very common pattern — GitHub code search returns tens of thousands of workflows doing it.
The one input that *would* be rejected is a ref containing a comma, but git refs with commas are pathological, this is a manually typed `workflow_dispatch` input, and such a ref would not resolve in `delta-io/delta` anyway — and the failure would be an explicit "cannot contain commas" message.
Against that, sanitizing has a real cost: it changes every existing cache key, so all shards would take a one-time full miss and re-download the sbt/Ivy/Coursier trees. Not worth it for a failure mode that doesn't exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3617401929
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,446 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib artifact it already built, so the expensive
+ # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a
+ # called workflow runs as part of the caller run), so the jobs below download it
+ # by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. `spark_version` is the single source of truth for the
+ # Spark version: it drives the Gluten bundle profile (-Pspark-<v>), the bundle
+ # jar name, and Delta's -DsparkVersion, so the tests always run against a bundle
+ # built for the same Spark version (no separate value to keep in sync). Scala
+ # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from
+ # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a
+ # compatible delta_ref.
+ GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version }}
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version }}
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib and passes it as an input, so this
+ # job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
Review Comment:
Same approach as other YAML files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3617389297
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,208 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# init + shallow fetch resolves a tag, branch OR commit SHA in a single path
+# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the
+# destructive `rm -rf "$DELTA_DIR"` it required. `--` terminates options so a
+# DELTA_REF starting with `-` can't be misread as a git flag (this script is
+# workflow_dispatch-runnable with a user-supplied ref).
+git init -q "$DELTA_DIR"
+git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git
+git -C "$DELTA_DIR" fetch -q --depth 1 origin -- "$DELTA_REF"
+git -C "$DELTA_DIR" checkout -q FETCH_HEAD
+git -C "$DELTA_DIR" --no-pager log -1 --oneline
+echo "::endgroup::"
+
+echo "::group::Injecting Gluten bundle jar onto the spark project's TEST classpath"
+# The Gluten bundle jar must be on the spark project's TEST runtime classpath
+# (so DeltaSQLCommandTest can load org.apache.gluten.GlutenPlugin by name) but
+# NOT on the COMPILE classpath of `sparkV1`, which is the project that holds
+# Delta's main sources. The bundle's transitive contents include extra symbols
+# under `org.apache.spark.sql` that collide with Delta's main sources -- e.g.
+# MergeOutputGeneration.scala imports both `org.apache.spark.sql._` and
+# `org.apache.spark.sql.delta.ClassicColumnConversions._`, and would then fail
+# with `reference to expression is ambiguous`.
+#
+# sbt auto-scans `<baseDirectory>/lib` via `unmanagedBase`. Two relevant
+# projects in Delta v4.2.0 have a `lib/` baseDirectory:
+# - sparkV1: `project in file("spark")` -> spark/lib
+# - spark : `project in file("spark-unified")` -> spark-unified/lib
+# unmanagedJars are project-scoped (NOT inherited by dependents), so dropping
+# the bundle into spark-unified/lib/ adds it to the unified `spark` project's
+# Compile *and* Test classpaths -- but NOT to sparkV1's. That's exactly what
+# we want:
+# * sparkV1/Compile sees ONLY Delta's regular deps -> Delta main compiles.
+# * spark/Test/fullClasspath sees the bundle -> tests load GlutenPlugin.
+# (Verified empirically: with bundle only in spark-unified/lib/, sbt's
+# `show sparkV1/Compile/dependencyClasspath` excludes the bundle and
+# `show spark/Test/fullClasspath` includes it.)
+#
+# We deliberately do NOT also drop the bundle into spark/lib/, which is what
+# caused the previous compile failure: spark/lib/ is sparkV1's unmanagedBase,
+# and putting the bundle there would re-introduce the ambiguity errors.
+SPARK_UNIFIED_LIB="$DELTA_DIR/spark-unified/lib"
+mkdir -p "$SPARK_UNIFIED_LIB"
+cp "$GLUTEN_BUNDLE_JAR" "$SPARK_UNIFIED_LIB/gluten-velox-bundle.jar"
+ls -lh "$SPARK_UNIFIED_LIB"
+echo "::endgroup::"
+
+echo "::group::Patching DeltaSQLCommandTest to enable Gluten plugin"
+TARGET="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$TARGET" ]; then
+ echo "Expected file not found in Delta clone: $TARGET" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
+ exit 1
+fi
+cp "$PATCH_SOURCE" "$TARGET"
+echo "Patched $TARGET"
+echo "--- diff vs. upstream ---"
+git -C "$DELTA_DIR" --no-pager diff -- "spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala" || true
+echo "::endgroup::"
+
+# Delta's tests collect file-source scans by matching the concrete
+# `FileSourceScanExec` case class; Gluten offloads the scan to
+# DeltaScanTransformer, a `FileSourceScanLike` sibling, so those matches miss
+# (`scala.MatchError: List()`, empty partition filters, broken column-pruning /
+# scan-metric checks across many suites). delta-io/delta#7104 and #7105 widen the
+# matches to the shared `FileSourceScanLike` interface that both the vanilla and
+# Gluten scans implement (behavior-preserving for vanilla). Both are merged
+# upstream but land after the pinned DELTA_REF (v4.2.0), so apply them here; once
+# DELTA_REF includes a commit its cherry-pick is a clean no-op and the call can go.
+#
+# Depth-2 fetch brings each fix commit and its parent, which cherry-pick needs to
+# diff against (a depth-1 fetch grafts the parent away); `-n` stages the change
+# without requiring a committer identity.
+cherry_pick_delta_fix() {
+ local sha="$1" pr="$2"
+ echo "Cherry-picking delta-io/delta${pr}"
+ git -C "$DELTA_DIR" fetch --quiet --depth 2 origin "$sha"
+ git -C "$DELTA_DIR" cherry-pick -n "$sha"
+}
+
+echo "::group::Cherry-picking upstream Delta FileSourceScanLike test fixes"
+cherry_pick_delta_fix 46bd45d57eadd7e528002a0ae7bd36ce5a456eca "#7104 (ScanReportHelper.collectScans)"
+cherry_pick_delta_fix 959e00e15f41f56afc1c9bb95d160c55c6dc7068 "#7105 (9 more test suites)"
+echo "::endgroup::"
+
+echo "::group::Force-failing memory-hog DeletionVectorsSuite 2B-row tests"
+# Two DeletionVectorsSuite tests read from / delete from a 2-billion-row table.
+# Under the Gluten Velox bundle they balloon the forked test JVM to ~13G of
+# NATIVE memory (row-index materialization) and the kernel/cgroup OOM-kills it.
+# The dead fork then wedges sbt, hanging the whole shard until the workflow's
+# hang-watchdog dumps threads and kills it (~16 min wasted, and every suite
+# QUEUED AFTER it in that fork is skipped) -- see delta_spark_ut.yml.
+#
+# Rather than silently `ignore` these (easy to forget), we make them FAIL FAST
+# with a clear message: the gap stays visible in the test reports / baseline
+# until the native memory blow-up is fixed, at which point this patch should be
+# removed. NOTE: making the suite complete also un-skips the rest of the shard's
+# suite queue, so the known-failures baseline must be refreshed after this.
+#
+# ORDER MATTERS: keep this sed AFTER the cherry-picks above. #7105 also edits
+# DeletionVectorsSuite.scala, and git cherry-pick aborts (exit 128) when the work
+# tree has uncommitted edits to a file it touches.
+DVS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala"
+if [ ! -f "$DVS" ]; then
+ echo "Expected file not found in Delta clone: $DVS" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed." >&2
+ exit 1
+fi
+# Inject `fail(...)` as the first statement of each test body (the line ending
+# in `) {`). Delta sets no -Xfatal-warnings / dead-code warning, so the now-
+# unreachable original body compiles fine. Keep each injected line <100 chars:
+# Delta's scalastyle enforces a 100-char line length on test sources. The full
+# rationale lives in this comment, so the in-test message stays terse.
+sed -i 's#huge table: read from tables of 2B rows with existing DV of many zeros") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV read OOMs the test JVM; see setup-delta.sh")#' "$DVS"
+sed -i 's#number of rows from tables of 2B rows with DVs") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV delete OOMs the test JVM; see setup-delta.sh")#' "$DVS"
+INJECTED=$(grep -c "Gluten CI] Force-failed" "$DVS" || true)
Review Comment:
Out of scope
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,208 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# init + shallow fetch resolves a tag, branch OR commit SHA in a single path
+# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the
+# destructive `rm -rf "$DELTA_DIR"` it required. `--` terminates options so a
+# DELTA_REF starting with `-` can't be misread as a git flag (this script is
+# workflow_dispatch-runnable with a user-supplied ref).
+git init -q "$DELTA_DIR"
+git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git
+git -C "$DELTA_DIR" fetch -q --depth 1 origin -- "$DELTA_REF"
+git -C "$DELTA_DIR" checkout -q FETCH_HEAD
+git -C "$DELTA_DIR" --no-pager log -1 --oneline
+echo "::endgroup::"
+
+echo "::group::Injecting Gluten bundle jar onto the spark project's TEST classpath"
+# The Gluten bundle jar must be on the spark project's TEST runtime classpath
+# (so DeltaSQLCommandTest can load org.apache.gluten.GlutenPlugin by name) but
+# NOT on the COMPILE classpath of `sparkV1`, which is the project that holds
+# Delta's main sources. The bundle's transitive contents include extra symbols
+# under `org.apache.spark.sql` that collide with Delta's main sources -- e.g.
+# MergeOutputGeneration.scala imports both `org.apache.spark.sql._` and
+# `org.apache.spark.sql.delta.ClassicColumnConversions._`, and would then fail
+# with `reference to expression is ambiguous`.
+#
+# sbt auto-scans `<baseDirectory>/lib` via `unmanagedBase`. Two relevant
+# projects in Delta v4.2.0 have a `lib/` baseDirectory:
+# - sparkV1: `project in file("spark")` -> spark/lib
+# - spark : `project in file("spark-unified")` -> spark-unified/lib
+# unmanagedJars are project-scoped (NOT inherited by dependents), so dropping
+# the bundle into spark-unified/lib/ adds it to the unified `spark` project's
+# Compile *and* Test classpaths -- but NOT to sparkV1's. That's exactly what
+# we want:
+# * sparkV1/Compile sees ONLY Delta's regular deps -> Delta main compiles.
+# * spark/Test/fullClasspath sees the bundle -> tests load GlutenPlugin.
+# (Verified empirically: with bundle only in spark-unified/lib/, sbt's
+# `show sparkV1/Compile/dependencyClasspath` excludes the bundle and
+# `show spark/Test/fullClasspath` includes it.)
+#
+# We deliberately do NOT also drop the bundle into spark/lib/, which is what
+# caused the previous compile failure: spark/lib/ is sparkV1's unmanagedBase,
+# and putting the bundle there would re-introduce the ambiguity errors.
+SPARK_UNIFIED_LIB="$DELTA_DIR/spark-unified/lib"
+mkdir -p "$SPARK_UNIFIED_LIB"
+cp "$GLUTEN_BUNDLE_JAR" "$SPARK_UNIFIED_LIB/gluten-velox-bundle.jar"
+ls -lh "$SPARK_UNIFIED_LIB"
+echo "::endgroup::"
+
+echo "::group::Patching DeltaSQLCommandTest to enable Gluten plugin"
+TARGET="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$TARGET" ]; then
+ echo "Expected file not found in Delta clone: $TARGET" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
+ exit 1
+fi
+cp "$PATCH_SOURCE" "$TARGET"
+echo "Patched $TARGET"
+echo "--- diff vs. upstream ---"
+git -C "$DELTA_DIR" --no-pager diff -- "spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala" || true
+echo "::endgroup::"
+
+# Delta's tests collect file-source scans by matching the concrete
+# `FileSourceScanExec` case class; Gluten offloads the scan to
+# DeltaScanTransformer, a `FileSourceScanLike` sibling, so those matches miss
+# (`scala.MatchError: List()`, empty partition filters, broken column-pruning /
+# scan-metric checks across many suites). delta-io/delta#7104 and #7105 widen the
+# matches to the shared `FileSourceScanLike` interface that both the vanilla and
+# Gluten scans implement (behavior-preserving for vanilla). Both are merged
+# upstream but land after the pinned DELTA_REF (v4.2.0), so apply them here; once
+# DELTA_REF includes a commit its cherry-pick is a clean no-op and the call can go.
+#
+# Depth-2 fetch brings each fix commit and its parent, which cherry-pick needs to
+# diff against (a depth-1 fetch grafts the parent away); `-n` stages the change
+# without requiring a committer identity.
+cherry_pick_delta_fix() {
+ local sha="$1" pr="$2"
+ echo "Cherry-picking delta-io/delta${pr}"
+ git -C "$DELTA_DIR" fetch --quiet --depth 2 origin "$sha"
+ git -C "$DELTA_DIR" cherry-pick -n "$sha"
+}
+
+echo "::group::Cherry-picking upstream Delta FileSourceScanLike test fixes"
+cherry_pick_delta_fix 46bd45d57eadd7e528002a0ae7bd36ce5a456eca "#7104 (ScanReportHelper.collectScans)"
+cherry_pick_delta_fix 959e00e15f41f56afc1c9bb95d160c55c6dc7068 "#7105 (9 more test suites)"
+echo "::endgroup::"
+
+echo "::group::Force-failing memory-hog DeletionVectorsSuite 2B-row tests"
+# Two DeletionVectorsSuite tests read from / delete from a 2-billion-row table.
+# Under the Gluten Velox bundle they balloon the forked test JVM to ~13G of
+# NATIVE memory (row-index materialization) and the kernel/cgroup OOM-kills it.
+# The dead fork then wedges sbt, hanging the whole shard until the workflow's
+# hang-watchdog dumps threads and kills it (~16 min wasted, and every suite
+# QUEUED AFTER it in that fork is skipped) -- see delta_spark_ut.yml.
+#
+# Rather than silently `ignore` these (easy to forget), we make them FAIL FAST
+# with a clear message: the gap stays visible in the test reports / baseline
+# until the native memory blow-up is fixed, at which point this patch should be
+# removed. NOTE: making the suite complete also un-skips the rest of the shard's
+# suite queue, so the known-failures baseline must be refreshed after this.
+#
+# ORDER MATTERS: keep this sed AFTER the cherry-picks above. #7105 also edits
+# DeletionVectorsSuite.scala, and git cherry-pick aborts (exit 128) when the work
+# tree has uncommitted edits to a file it touches.
+DVS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala"
+if [ ! -f "$DVS" ]; then
+ echo "Expected file not found in Delta clone: $DVS" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed." >&2
+ exit 1
+fi
+# Inject `fail(...)` as the first statement of each test body (the line ending
+# in `) {`). Delta sets no -Xfatal-warnings / dead-code warning, so the now-
+# unreachable original body compiles fine. Keep each injected line <100 chars:
+# Delta's scalastyle enforces a 100-char line length on test sources. The full
+# rationale lives in this comment, so the in-test message stays terse.
+sed -i 's#huge table: read from tables of 2B rows with existing DV of many zeros") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV read OOMs the test JVM; see setup-delta.sh")#' "$DVS"
+sed -i 's#number of rows from tables of 2B rows with DVs") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV delete OOMs the test JVM; see setup-delta.sh")#' "$DVS"
+INJECTED=$(grep -c "Gluten CI] Force-failed" "$DVS" || true)
+if [ "$INJECTED" -ne 2 ]; then
+ echo "ERROR: expected to force-fail 2 DeletionVectorsSuite tests but injected ${INJECTED}." >&2
+ echo "Their test names likely changed in Delta ref '${DELTA_REF}'; update setup-delta.sh." >&2
+ exit 1
+fi
+echo "Force-failed 2 DeletionVectorsSuite 2B-row tests (read + delete)."
+git -C "$DELTA_DIR" --no-pager diff -- "spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala" || true
+echo "::endgroup::"
+
+echo "::group::Disabling Delta scalastyle HeaderMatchesChecker"
+# Our reused DeltaSQLCommandTest carries Gluten's ASF-only license header, which
+# does not match Delta's HeaderMatchesChecker regex (the regex expects either a
+# Delta copyright block, or the ASF header followed by a Spark-modifications
+# block and the Delta copyright block). HeaderMatchesChecker is a file-level
+# checker that does NOT honor `// scalastyle:off` directives, so we instead
+# disable it globally in Delta's shared scalastyle-config.xml. The config is
+# applied via `ThisBuild / scalastyleConfig` in project/Checkstyle.scala, so a
+# single edit covers every sbt sub-project.
+SCALASTYLE_CONFIG="$DELTA_DIR/scalastyle-config.xml"
+if [ ! -f "$SCALASTYLE_CONFIG" ]; then
+ echo "Expected scalastyle config not found: $SCALASTYLE_CONFIG" >&2
+ exit 1
+fi
+sed -i \
+ 's|<check level="error" class="org.scalastyle.file.HeaderMatchesChecker" enabled="true">|<check level="error" class="org.scalastyle.file.HeaderMatchesChecker" enabled="false">|' \
+ "$SCALASTYLE_CONFIG"
+if ! grep -q '<check level="error" class="org.scalastyle.file.HeaderMatchesChecker" enabled="false">' "$SCALASTYLE_CONFIG"; then
Review Comment:
Out of scope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3532698563
##########
.github/workflows/util/delta-spark-ut/known-failures.txt:
##########
@@ -0,0 +1,830 @@
+# Known Delta-on-Gluten unit test failures.
+#
+# Baseline of delta-io/delta `spark` ScalaTest tests EXPECTED to fail under the
+# Gluten Velox bundle. The Delta Spark UT (Gluten) workflow enforces this list:
+# a failing test NOT listed here is a regression (fails CI); a listed test that
+# now passes should be removed. Format: <fully.qualified.SuiteName>#<test name>.
+# Lines starting with '#' are comments. See README.md in this directory.
+#
+# ---------------------------------------------------------------------------
+# Baseline for the committed 4-shard x 4-fork config. Originally seeded from run
+# 27490052632; refreshed from run 28318129710 (4 x 4) after the FileSourceScanLike
+# test fixes (delta-io/delta #7104 + #7105). 811 known failures total.
Review Comment:
dropped the hard-coded total
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3533057511
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -19,6 +19,11 @@ on:
pull_request:
paths:
- '.github/workflows/velox_backend_x86.yml'
+ # Delta Spark UT runs here too (reusable delta_spark_ut.yml). These extra
+ # paths make Delta-CI-only changes trigger this workflow; Delta also runs on
+ # the velox paths below since core/velox changes can affect Delta offload.
+ - '.github/workflows/delta_spark_ut.yml'
+ - '.github/workflows/util/delta-spark-ut/**'
Review Comment:
Intentional for now: core/velox/substrait changes can affect Delta offload so Delta shares this workflow's native build, and the extra matrix cost on a baseline-only edit is an accepted tradeoff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3518962952
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,509 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline file exists but is empty (not yet bootstrapped) the mode
+ automatically degrades to ``seed`` so the first run is never spuriously red.
+ A *missing* ``--known-failures`` file is treated as a configuration error
+ (the gate fails) so a mis-referenced path can't silently pass.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard).
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import glob
+import os
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+
+class NoReportsError(RuntimeError):
+ """Raised when no JUnit <testsuite> elements are found under reports_dir."""
+
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped) sets of (suite, test) tuples. A test is
+ 'failed' if its <testcase> has a <failure> or <error> child, 'skipped' if
+ it has a <skipped> child, otherwise 'passed'. Suite-level aborts (a
+ <testsuite> reporting errors/failures with no failing <testcase>) are
+ recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
Review Comment:
`parse_reports()` logs and skips XML parse errors. If a JUnit report is truncated/corrupted (e.g., forked JVM killed mid-write), silently ignoring it can drop failures from the evaluated result set and potentially miss regressions. For this gate, it’s safer to fail fast at least for expected report files (TEST-*.xml / surefire-style reports) so CI can’t go green on partial data.
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,683 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Delta `-DsparkVersion` value (must match the Gluten -P profile below)'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. Spark 4.1 + Scala 2.13 + JDK 17 matches Delta v4.2.0's
+ # default Spark version (4.1.0) from project/CrossSparkVersions.scala.
+ GLUTEN_SPARK_PROFILE: 'spark-4.1'
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: '4.1'
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Every input has a default (workflow_call + workflow_dispatch), so they
+ # are always set; just surface them as step outputs for the steps below.
+ env:
+ DELTA_REF: ${{ inputs.delta_ref }}
+ SPARK_VERSION: ${{ inputs.spark_version }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline }}
+ FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
+
+ - name: Download Gluten bundle jar
+ uses: actions/download-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: gluten-bundle
+
+ - name: Install minimal tools
+ run: |
+ set -euo pipefail
+ # apache/gluten:centos-9-jdk17 already has java-17, git, tar, a POSIX
+ # shell, and curl-minimal (which provides the `curl` command sbt's
+ # launcher needs). Install the rest of what Delta's build/sbt and the
+ # tests may need. We deliberately do NOT install the full `curl`
+ # package -- it conflicts with the pre-installed curl-minimal.
+ yum install -y java-17-openjdk-devel which findutils gzip python3
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ git --version
+ curl --version | head -n 1
+
+ - name: Cache sbt / Ivy / Coursier
+ uses: actions/cache@v4
+ with:
+ path: |
+ /root/.sbt
+ /root/.ivy2
+ /root/.cache/coursier
+ # Intentionally NOT keyed by ${{ matrix.shard }} -- all shards share
+ # the same dependency tree, so a single shared cache (with parallel
+ # save races resolved by GH on a first-write-wins basis) gives the
+ # best storage / hit-rate tradeoff.
+ key: delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-${{ env.DELTA_SCALA_VERSION }}
+ restore-keys: |
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-
+
+ - name: Clone and patch Delta
+ run: |
+ set -euo pipefail
+ # `2>/dev/null ... || true` keeps a no-match `ls` from aborting the step
+ # under `set -o pipefail`, so the explicit check below emits a clear
+ # error instead of a generic "cannot access".
+ GLUTEN_JAR=$(ls "$GITHUB_WORKSPACE"/gluten-bundle/gluten-velox-bundle-spark*_*-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$GLUTEN_JAR" ] || [ ! -f "$GLUTEN_JAR" ]; then
+ echo "ERROR: No Gluten bundle jar found under $GITHUB_WORKSPACE/gluten-bundle/" >&2
+ ls -la "$GITHUB_WORKSPACE/gluten-bundle/" || true
+ exit 1
+ fi
+ echo "Using Gluten bundle: $GLUTEN_JAR"
+ bash "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/setup-delta.sh" \
+ "${{ steps.resolve.outputs.delta_ref }}" \
+ "$GITHUB_WORKSPACE/delta" \
+ "$GLUTEN_JAR" \
+ "$GITHUB_WORKSPACE"
+
+ - name: Run Delta spark module tests (shard ${{ matrix.shard }} / ${{ env.DELTA_NUM_SHARDS }})
+ env:
+ NUM_SHARDS: ${{ env.DELTA_NUM_SHARDS }}
+ TEST_PARALLELISM_COUNT: ${{ steps.resolve.outputs.test_parallelism }}
+ # Required by Delta to enable testing-only code paths
+ # (see delta build.sbt: "Test / envVars += DELTA_TESTING -> 1").
+ DELTA_TESTING: '1'
+ # JDK 17 + Gluten/Arrow/Netty requires extra --add-opens and the
+ # `io.netty.tryReflectionSetAccessible` system property; otherwise
+ # the forked test JVM fails with
+ # java.lang.UnsupportedOperationException: sun.misc.Unsafe or
+ # java.nio.DirectByteBuffer.<init>(long, int) not available
+ # as soon as Gluten's bundled Arrow allocator initializes Netty
+ # direct buffers. Delta's own `Test / javaOptions` (see
+ # project/CrossSparkVersions.scala `java17TestSettings`) sets the
+ # base add-opens but NOT the Netty property -- Delta's own tests
+ # don't load Arrow/Netty buffers in a way that triggers it.
+ #
+ # Use JAVA_TOOL_OPTIONS so the flags propagate to BOTH the sbt
+ # launcher JVM and the forked test JVM (sbt forks tests and the
+ # child inherits the parent's env). The set below mirrors
+ # `extraJavaTestArgs` from Gluten's own root pom.xml (the
+ # canonical Gluten test JVM flag set).
+ #
+ # NOTE: we deliberately do NOT put `-Xmx` here. JAVA_TOOL_OPTIONS
+ # is processed BEFORE the JVM command line, so Delta's explicit
+ # `-Xmx1024m` (set in build.sbt `Test / javaOptions`) would still
+ # win (last `-Xmx` wins). The forked-test-JVM heap is bumped via
+ # an sbt `set spark / Test / javaOptions ++= ...` command below,
+ # which APPENDS to Delta's own seq -- so our `-Xmx` lands AFTER
+ # `-Xmx1024m` and wins.
+ JAVA_TOOL_OPTIONS: >-
+ -XX:+IgnoreUnrecognizedVMOptions
+ --add-opens=java.base/java.lang=ALL-UNNAMED
+ --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
+ --add-opens=java.base/java.lang.reflect=ALL-UNNAMED
+ --add-opens=java.base/java.io=ALL-UNNAMED
+ --add-opens=java.base/java.net=ALL-UNNAMED
+ --add-opens=java.base/java.nio=ALL-UNNAMED
+ --add-opens=java.base/java.util=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
+ --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.cs=ALL-UNNAMED
+ --add-opens=java.base/sun.security.action=ALL-UNNAMED
+ --add-opens=java.base/sun.util.calendar=ALL-UNNAMED
+ -Djdk.reflect.useDirectMethodHandle=false
+ -Dio.netty.tryReflectionSetAccessible=true
+ -Dfile.encoding=UTF-8
+ run: |
+ set -euo pipefail
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ cd "$GITHUB_WORKSPACE/delta"
+ chmod +x build/sbt
+ # Only run the unified `spark` sbt project, NOT `sparkGroup/test` --
+ # `sparkGroup` aggregates many other projects (sparkV2, contribs,
+ # sharing, connect*, ...) that are out of scope for this pipeline.
+ #
+ # JVM heap layout -- two memory consumers on the ~16G runner:
+ # * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to
+ # return idle memory during the (long) test phase via G1 periodic GC
+ # (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the
+ # busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent
+ # forces each periodic GC to a full STW collection) that uncommits to a
+ # tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low
+ # -Xms512m floor.
+ # Without this the idle launcher holds ~5.3G for the whole run; with
+ # it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime
+ # config, so they cannot affect the measured pass/fail signal.
+ # * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command
+ # below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends
+ # so our -Xmx2G comes last and wins. Gluten offloads data to Velox
+ # off-heap (capped at 2g via spark.memory.offHeap.size in the patched
+ # DeltaSQLCommandTest), so the fork's heap need is modest. A larger
+ # fork heap pushed the cgroup peak past the ~16G OOM threshold and the
+ # kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G
+ # keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is
+ # analyzable.
+ # `-u target/test-reports` enables ScalaTest's JUnit XML reporter so
+ # every suite writes per-test results. Delta itself only configures
+ # the console reporter (-oDF), so without this we'd have no machine-
+ # readable results to gate on. The path is relative to the forked
+ # test JVM's working dir (Test / baseDirectory = spark/), i.e.
+ # delta/spark/target/test-reports/TEST-*.xml.
+ #
+ # We deliberately do NOT let an sbt non-zero exit (which fires on the
+ # MANY expected Delta-on-Gluten failures) fail this step directly.
+ # Instead the known-failures gate below decides pass/fail: the build
+ # is green when the only failures are ones already recorded in the
+ # baseline, and red on a genuine regression.
+ set +e
+ # --- hang watchdog ---------------------------------------------------
+ # Shard 2 (and occasionally others) hangs indefinitely after a suite's
+ # last test with no further output. ScalaTest's failAfter only wraps
+ # individual test BODIES, so a wedge in suite teardown/afterAll -- or in
+ # a non-interruptible native Velox/JNI call that ignores
+ # Thread.interrupt() -- has no timeout and stalls until the 350-min job
+ # limit with zero diagnostics. This watchdog dumps the forked test JVM's
+ # threads (to the job log, and to a file for the artifact) once the test
+ # output has been silent for too long, so the deadlock is diagnosable.
+ SBT_LOG="/tmp/sbt-spark-test-shard-${{ matrix.shard }}.log"
+ : > "$SBT_LOG"
+ rm -f /tmp/sbt-done
+ (
+ # CRITICAL: the step shell runs with `bash -eo pipefail`, which the
+ # subshell inherits. Without `set +e` here, ANY non-zero command --
+ # e.g. fork detection finding no match, or `kill`/`jps` returning
+ # non-zero -- silently kills this watchdog. That errexit kill (plus a
+ # /proc detection miss) once made the watchdog capture ZERO dumps. A
+ # diagnostic must never abort on a failed probe.
+ set +e +o pipefail
+ JSTACK="${JAVA_HOME}/bin/jstack"
+ JPS="${JAVA_HOME}/bin/jps"
+ silent_limit=900 # 15 min with no new test output => treat as hung
+ dumps=0
+ fork_pids() {
+ # The sbt test fork's main class is sbt.ForkMain. Prefer jps (reads
+ # the main class from hsperfdata, robust to sbt's @argfile launch);
+ # fall back to scanning /proc cmdline + @argfile.
+ "$JPS" -l 2>/dev/null | awk '/sbt\.ForkMain/ {print $1}'
+ local p cl arg
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] || continue
+ cl="$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)"
+ case "$cl" in *sbt.ForkMain*) echo "${p##*/}"; continue ;; esac
+ arg="$(printf '%s' "$cl" | tr ' ' '\n' | sed -n 's/^@//p' | head -1)"
+ [ -n "$arg" ] && [ -f "$arg" ] && grep -qa 'sbt\.ForkMain' "$arg" 2>/dev/null \
+ && echo "${p##*/}"
+ done
+ }
+ all_java_pids() {
+ "$JPS" -q 2>/dev/null
+ local p
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] && echo "${p##*/}"
+ done
+ }
+ echo "HANG WATCHDOG armed: dumps the test JVM after ${silent_limit}s of output silence"
+ hb=0
+ while [ ! -f /tmp/sbt-done ]; do
+ sleep 60
+ [ -f "$SBT_LOG" ] || continue
+ now=$(date +%s)
+ mtime=$(stat -c %Y "$SBT_LOG" 2>/dev/null || echo "$now")
+ silent=$(( now - mtime ))
+ # Per-minute memory profile: heap tuning proved the ~16G OOM peak is
+ # NATIVE-driven, so log which JVM (sbt launcher vs fork) actually grows
+ # toward it -- the last lines before a hang reveal the real hog to cut.
+ # Read /proc directly (no `ps` dependency in the minimal container).
+ memnow=$(awk '{printf "%.2fG",$1/1073741824}' /sys/fs/cgroup/memory.current 2>/dev/null)
+ jvmrss=""
+ for mp in $(all_java_pids 2>/dev/null | sort -un); do
+ r=$(awk '/^VmRSS:/{print $2}' "/proc/$mp/status" 2>/dev/null)
+ [ -n "$r" ] && jvmrss="$jvmrss $(( r / 1024 ))M(p$mp)"
+ done
+ echo "MEM cgroup=${memnow} JVMs=[${jvmrss# }]"
+ hb=$(( hb + 1 ))
+ # Heartbeat every ~5 min so we can SEE the watchdog is alive (and how
+ # long the test has been silent) without waiting for a hang.
+ [ $(( hb % 5 )) -eq 0 ] && echo "HANG WATCHDOG: alive; last test output ${silent}s ago"
+ if [ "$silent" -ge "$silent_limit" ] && [ "$dumps" -lt 3 ]; then
+ dumps=$(( dumps + 1 ))
+ pids="$(fork_pids | sort -un)"
+ # Safety net: if the fork JVM cannot be pinpointed, dump EVERY JVM.
+ [ -n "$pids" ] || pids="$(all_java_pids | sort -un)"
+ echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $pids))"
+ [ -n "$pids" ] || echo "HANG WATCHDOG: no java process found to dump"
+ for pid in $pids; do
+ # SIGQUIT makes the JVM print a full thread dump to its OWN stderr,
+ # which sbt relays into the test log via the SAME stream as test
+ # output -- so it lands in the job log even when a separately
+ # spawned jstack child's output would be buffered/lost. Also write
+ # jstack to a file for the per-shard artifact.
+ echo "----- SIGQUIT + jstack pid ${pid} -----"
+ kill -QUIT "$pid" 2>/dev/null || echo "HANG WATCHDOG: kill -QUIT failed for pid ${pid}"
+ timeout 120 "$JSTACK" -l "$pid" > "/tmp/threaddump-shard-${{ matrix.shard }}-${dumps}-${pid}.txt" 2>&1 \
+ || echo "HANG WATCHDOG: jstack failed/timed out for pid ${pid}"
+ done
+ echo "::endgroup::"
+ # The dump is now captured (job log via SIGQUIT + artifact via
+ # jstack file). A hung JVM otherwise stalls the whole shard until
+ # the 350-min job timeout AND keeps the job log frozen so the dump
+ # never becomes reachable. So KILL the wedged JVM(s): the suite
+ # fails fast (acceptable -- errors are expected; only an
+ # unrecoverable hang blocks CI), the job proceeds/ends, and the log
+ # + artifacts flush. Give SIGQUIT a moment to print first.
+ sleep 20
+ echo "HANG WATCHDOG: killing wedged JVM(s) to unblock the shard: $(printf '%s ' $pids)"
+ for pid in $pids; do kill -KILL "$pid" 2>/dev/null; done
+ fi
+ done
+ ) &
+ WATCHDOG_PID=$!
+
+ ./build/sbt \
+ -DsparkVersion=${{ steps.resolve.outputs.spark_version }} \
+ -v \
+ -J-XX:+UseG1GC -J-Xms512m -J-Xmx4G \
+ -J-XX:G1PeriodicGCInterval=10000 \
+ -J-XX:G1PeriodicGCSystemLoadThreshold=0 \
+ -J-XX:-G1PeriodicGCInvokesConcurrent \
+ -J-XX:MinHeapFreeRatio=5 -J-XX:MaxHeapFreeRatio=15 \
+ "++ ${DELTA_SCALA_VERSION}" \
+ 'set spark / Test / javaOptions ++= Seq("-Xmx2G", "-XX:+HeapDumpOnOutOfMemoryError", "-XX:HeapDumpPath=/tmp/")' \
+ 'set spark / Test / testOptions += Tests.Argument(TestFrameworks.ScalaTest, "-u", "target/test-reports")' \
+ "spark/test" 2>&1 | tee "$SBT_LOG"
+ SBT_EXIT=${PIPESTATUS[0]}
+ touch /tmp/sbt-done
+ kill "$WATCHDOG_PID" 2>/dev/null || true
+ set -e
+ echo "sbt spark/test exited with ${SBT_EXIT}"
+
+ # Memory forensics: a sudden forked-JVM death with no hs_err and no heap
+ # dump is almost always a kernel/cgroup OOM-kill (Velox off-heap + JVM
+ # heap exceeding the ~16G runner). Surface the cgroup peak + oom_kill
+ # count so we can confirm/measure it (cgroup v2 paths; best-effort).
+ ( echo "=== cgroup memory forensics (exit ${SBT_EXIT}) ==="
+ for f in /sys/fs/cgroup/memory.peak /sys/fs/cgroup/memory.max \
+ /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.events; do
+ [ -r "$f" ] && { echo "--- $f ---"; cat "$f"; }
+ done ) || true
+
+ # A compile/launch failure leaves no reports at all. In that case the
+ # gate would see zero failures and pass spuriously, so fail loudly.
+ REPORT_COUNT=$(find . -path '*/target/test-reports/*.xml' 2>/dev/null | wc -l || true)
+ echo "Found ${REPORT_COUNT} JUnit XML report file(s)."
+ if [ "${REPORT_COUNT}" -eq 0 ]; then
+ echo "::error::sbt produced no test reports (exit ${SBT_EXIT}) -- likely a compile or launch failure, not test failures."
+ exit 1
+ fi
+
+ # update_baseline=true -> SEED mode (record failures, never fail) so the
+ # baseline can be (re)generated. Otherwise ENFORCE against the baseline.
+ GATE_MODE=enforce
+ if [ "${{ steps.resolve.outputs.update_baseline }}" = "true" ]; then
+ GATE_MODE=seed
+ fi
+ mkdir -p "$GITHUB_WORKSPACE/gate-out"
+ python3 "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/compare-test-results.py" \
+ --mode "${GATE_MODE}" \
+ --reports-dir "$GITHUB_WORKSPACE/delta" \
+ --known-failures "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/known-failures.txt" \
+ --failures-out "$GITHUB_WORKSPACE/gate-out/failures-shard-${{ matrix.shard }}.txt" \
+ --ran-out "$GITHUB_WORKSPACE/gate-out/ran-shard-${{ matrix.shard }}.txt" \
+ --fail-on-fixed "${{ steps.resolve.outputs.fail_on_fixed }}"
+
+ - name: Compress heap dumps (if any)
+ if: ${{ failure() }}
+ run: |
+ set -euo pipefail
+ if compgen -G "/tmp/*.hprof" > /dev/null; then
+ echo "Found heap dump(s); compressing..."
+ ls -lh /tmp/*.hprof
+ # gzip is single-threaded and slow on multi-GB heaps but is
+ # always present in the centos image. Heap dumps compress ~10x.
+ gzip -1 /tmp/*.hprof
+ ls -lh /tmp/*.hprof.gz
+ else
+ echo "No heap dumps found in /tmp/."
+ fi
+
+ - name: Upload per-shard gate lists
+ if: always()
+ uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gate-lists-shard-${{ matrix.shard }}
+ path: gate-out/*.txt
+ if-no-files-found: warn
+
+ - name: Upload test reports
+ if: always()
+ uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-reports-shard-${{ matrix.shard }}
+ path: |
+ delta/**/target/test-reports/**/*.xml
+ delta/**/target/surefire-reports/**/*.xml
+ if-no-files-found: warn
+
+ - name: Upload hang watchdog thread dumps
+ if: always()
+ uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-threaddumps-shard-${{ matrix.shard }}
+ path: /tmp/threaddump-shard-${{ matrix.shard }}-*.txt
+ if-no-files-found: ignore
+
+ - name: Upload JVM crash logs and other failure artifacts
+ if: ${{ failure() }}
+ uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-failure-logs-shard-${{ matrix.shard }}
+ path: |
+ delta/**/target/*.log
+ delta/**/hs_err_pid*.log
+ delta/**/core.*
+ /tmp/*.hprof
+ /tmp/*.hprof.gz
+ if-no-files-found: ignore
+
+ # Merges every shard's failure/ran lists into a single, sorted, ready-to-commit
+ # known-failures.txt and reports global regressions / now-passing / stale
+ # entries. Runs even when some shards went red (if: always()) so the refreshed
+ # baseline artifact is always available -- this is what you download and commit
+ # to bootstrap or refresh the baseline (see util/delta-spark-ut/README.md).
+ delta-spark-aggregate:
+ needs: delta-spark-test
+ if: always()
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download per-shard gate lists
+ uses: actions/download-artifact@v4
+ continue-on-error: true
+ with:
+ pattern: delta-spark-ut-gate-lists-shard-*
+ path: gate-lists
+ merge-multiple: true
Review Comment:
The aggregate job downloads gate-list artifacts with `continue-on-error: true`, and the aggregator script only hard-fails when *no* inputs are found. If some shard artifacts are missing (or only partially downloaded), the job can still produce an "aggregated/known-failures.txt" that looks ready-to-commit but is incomplete, which can accidentally shrink the baseline and create noisy follow-up CI failures. Consider validating that gate lists were collected for every shard (e.g., pass an expected shard count to `compare-test-results.py --mode aggregate` and fail when fewer/mismatched shard inputs are present).
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -19,6 +19,11 @@ on:
pull_request:
paths:
- '.github/workflows/velox_backend_x86.yml'
+ # Delta Spark UT runs here too (reusable delta_spark_ut.yml). These extra
+ # paths make Delta-CI-only changes trigger this workflow; Delta also runs on
+ # the velox paths below since core/velox changes can affect Delta offload.
+ - '.github/workflows/delta_spark_ut.yml'
+ - '.github/workflows/util/delta-spark-ut/**'
Review Comment:
Adding Delta-CI-only paths under `on.pull_request.paths` makes *the entire* `Velox Backend (x86)` workflow run when only the Delta UT workflow/baseline changes. Since this workflow contains additional expensive jobs (e.g., the Spark/TPC matrices), a baseline-only edit to `known-failures.txt` will now trigger the full suite rather than just Delta UT. If the intent is to validate only the Delta pipeline on Delta-CI file changes, consider adding a path-filtering pre-job and gating non-Delta jobs with an `if:` condition, or introducing a dedicated lightweight caller workflow that only invokes `delta_spark_ut.yml`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3531273456
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,672 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Delta `-DsparkVersion` value (must match the Gluten -P profile below)'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. Spark 4.1 + Scala 2.13 + JDK 17 matches Delta v4.2.0's
+ # default Spark version (4.1.0) from project/CrossSparkVersions.scala.
+ GLUTEN_SPARK_PROFILE: 'spark-4.1'
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: '4.1'
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3533115069
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,208 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
Review Comment:
Not needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3555568061
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,747 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline file exists but is empty (not yet bootstrapped) the mode
+ automatically degrades to ``seed`` so the first run is never spuriously red.
+ A *missing* ``--known-failures`` file is treated as a configuration error
+ (the gate fails) so a mis-referenced path can't silently pass.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard). Pass ``--expected-shards N``
+ to fail when fewer than ``N`` shards contributed gate lists (a shard that
+ died before writing them), so an incomplete baseline is never produced.
+
+Flaky quarantine (``--flaky-tests``)
+ Some Delta-on-Gluten failures are non-deterministic (e.g. a native bug that
+ only triggers on certain runtime plans), so they are neither a stable pass
+ nor a stable failure and cannot live in the baseline: baselining them turns
+ the gate red on every run where they pass, and leaving them out turns it red
+ on every run where they fail. ``flaky-tests.txt`` quarantines them -- a
+ quarantined test never counts as a regression (when it fails) nor as
+ now-passing (when it passes), and is excluded from the regenerated baseline.
+ Its SUITE is an fnmatch glob (so one line covers a root-cause family across
+ generated suite variants); its TEST name is matched exactly.
+
+Flaky quarantine by error signature (``--flaky-error-patterns``)
+ When a failure is caused by a known nondeterministic bug that surfaces on a
+ *different test each run* (e.g. the native Delta DV bitmap row-index error),
+ matching by test name is whack-a-mole. ``flaky-error-patterns.txt`` instead
+ quarantines by root cause: each line is a regex matched against a failed
+ test's <failure>/<error> text, and any failure that matches is treated as
+ flaky regardless of which test it landed on.
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import fnmatch
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+
+class NoReportsError(RuntimeError):
+ """Raised when no JUnit <testsuite> elements are found under reports_dir."""
+
+
+class CorruptReportError(NoReportsError):
+ """Raised when an expected JUnit report file (TEST-*.xml) fails to parse.
+
+ Subclasses NoReportsError so the enforce/seed handler treats a truncated
+ report as a hard data error (exit 2) instead of silently dropping the
+ suite's results and letting the gate pass on partial data.
+ """
+
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def make_is_flaky(flaky_entries):
+ """Build a predicate that matches a (suite, test) tuple against flaky entries.
+
+ A flaky entry quarantines a test whose failure is known to be non-deterministic
+ (see flaky-tests.txt). The entry's SUITE is treated as an fnmatch glob so a
+ single line can cover a root-cause family across generated suite variants
+ (e.g. ``*DVs*Suite`` matches every deletion-vector merge suite, ``*`` matches
+ any suite); the TEST name is matched exactly (test names are freeform and may
+ contain glob metacharacters, so they are never globbed).
+ """
+ exact = set()
+ globbed = []
+ for suite, test in flaky_entries:
+ if any(ch in suite for ch in "*?["):
+ globbed.append((suite, test))
+ else:
+ exact.add((suite, test))
+
+ def is_flaky(entry):
+ if entry in exact:
+ return True
+ suite, test = entry
+ for glob_suite, glob_test in globbed:
+ if test == glob_test and fnmatch.fnmatchcase(suite, glob_suite):
+ return True
+ return False
+
+ return is_flaky
+
+
+def load_patterns(path):
+ """Load flaky-error regex patterns from a file (one per line).
+
+ Blank lines and ``#`` comments are ignored. Each remaining line is compiled
+ as a case-sensitive regex. These match the FAILURE TEXT of a failed test
+ (its JUnit <failure>/<error> message + stack), so a test that fails with a
+ known-nondeterministic native error (e.g. the Delta DV bitmap row-index bug)
+ can be quarantined by root cause instead of by exact test name.
+ """
+ patterns = []
+ if not path or not os.path.exists(path):
+ return patterns
+ with open(path, encoding="utf-8") as fh:
+ for line in fh:
+ line = line.rstrip("\n")
+ if not line.strip() or line.lstrip().startswith("#"):
+ continue
+ patterns.append(re.compile(line))
+ return patterns
Review Comment:
`load_patterns()` compiles each non-comment line as a regex without any error context. If a contributor adds an invalid pattern to flaky-error-patterns.txt, the workflow will fail with an unhelpful Python stack trace that doesn’t identify which line/pattern was invalid. Catch `re.error` and rethrow with the file path + line number + pattern text so failures are actionable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-4971024085
> @felipepessoto could you please point me to the tests that you think might be failing due to TimestampNTZ? I'd like to take a closer look. It appears that the tests in the log you shared here [#11622 (comment)](https://github.com/apache/gluten/issues/11622#issuecomment-4965875346) finished without failure.
@rui-mo one example is at line 5706:
`[info] - data skipping on TIMESTAMP_NTZ - old behavior with DataFrame schema *** FAILED ***`
If you search in the logs by `Cause: java.lang.ClassCastException: class org.apache.spark.sql.execution.ProjectExec` you can find a few more cases. The CI is passing because I created a list of known issues that are ignored, or else the CI would always fail, we have around 800 failing tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3669339006
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -101,6 +106,80 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR
+ # touches high-signal Delta paths -- the Delta integration code
+ # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own
+ # files -- or carries the `run-delta-ci` opt-in label. Changes to general
+ # Velox/core/native code can also affect Delta offload but are touched
+ # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml
+ # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage
+ # down. NOTE: the label is read from the event that triggered this run, so add
+ # it before/with a push; labeling an already-finished run needs a new push.
+ delta-changes:
+ runs-on: ubuntu-22.04
+ outputs:
+ run_delta: ${{ steps.filter.outputs.run_delta }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ - name: Detect Delta-relevant changes / opt-in label
+ id: filter
+ env:
+ HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+ run: |
+ set -euo pipefail
+ # Opt-in label forces the suite even with no Delta-relevant path change.
+ if [ "$HAS_LABEL" = "true" ]; then
+ echo "run-delta-ci label present -> running Delta suite"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
+ fi
+ # Fail open if we can't determine the PR range (e.g. a non-PR trigger):
+ # never silently skip coverage.
+ if [ -z "${BASE_SHA:-}" ] || [ -z "${HEAD_SHA:-}" ]; then
+ echo "no PR base/head sha -> running Delta suite (fail-open)"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
+ fi
+ BASE=$(git merge-base "$BASE_SHA" "$HEAD_SHA" 2>/dev/null || echo "$BASE_SHA")
+ echo "diff base=$BASE head=$HEAD_SHA"
+ # Fail open if the diff itself can't be computed (missing objects after a
+ # force-push race, an unfetched fork head, ...). Piping straight into
+ # `grep -q` inside an `if` would hide that: git's failure leaves grep with
+ # empty input, so the pipeline exits non-zero exactly as it does for "no
+ # match" -- and `set -e`/`pipefail` can't help, since a tested command is
+ # allowed to fail. Capture the diff first so the two cases stay distinct.
+ if ! CHANGED=$(git diff --name-only "$BASE" "$HEAD_SHA"); then
Review Comment:
This PR is a fork PR ( felipepessoto/gluten → apache/gluten ), and across 4/4 runs delta-changes printed Delta-relevant paths changed , never git diff failed . The fork head is reachable because refs/pull/N/merge 's second parent is the fork head commit, so checkout brings it into the local object store.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5243418078
@felipepessoto thanks for the improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3617361282
##########
.github/workflows/util/delta-spark-ut/run-delta-tests.sh:
##########
@@ -0,0 +1,240 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Runs the Delta `spark` module tests for one shard under the Gluten bundle:
+# arms a hang watchdog (thread-dumps + kills a wedged fork), invokes sbt
+# spark/test with the tuned JVM/heap flags, prints cgroup memory forensics, and
+# then gates the results against the baseline (compare-test-results.py). Extracted
+# from delta_spark_ut.yml so the workflow step stays readable.
+#
+# Driven by environment (set by the workflow step / job):
+# SHARD_ID - this shard's id (matrix.shard)
+# SPARK_VERSION - Delta -DsparkVersion value
+# UPDATE_BASELINE - 'true' -> gate seed mode; else enforce
+# FAIL_ON_FIXED - passed through to the gate
+# DELTA_SCALA_VERSION, NUM_SHARDS, TEST_PARALLELISM_COUNT, DELTA_TESTING
+# - test env (see the workflow step's `env:` block)
+# GITHUB_WORKSPACE - repo root (holds the Delta clone + util scripts)
+#
+# JAVA_TOOL_OPTIONS is set by sourcing java-test-args.sh (below), not the caller.
+
+set -euo pipefail
+export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+export PATH=$JAVA_HOME/bin:$PATH
+# Gluten/JDK17 test JVM flags (--add-opens + Netty property), shared with local
+# dev runs. Sets JAVA_TOOL_OPTIONS so it reaches the sbt launcher + forked JVMs.
+# shellcheck source=./java-test-args.sh
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/java-test-args.sh"
+cd "$GITHUB_WORKSPACE/delta"
+chmod +x build/sbt
+# Only run the unified `spark` sbt project, NOT `sparkGroup/test` --
+# `sparkGroup` aggregates many other projects (sparkV2, contribs,
+# sharing, connect*, ...) that are out of scope for this pipeline.
+#
+# JVM heap layout -- two memory consumers on the ~16G runner:
+# * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to
+# return idle memory during the (long) test phase via G1 periodic GC
+# (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the
+# busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent
+# forces each periodic GC to a full STW collection) that uncommits to a
+# tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low
+# -Xms512m floor.
+# Without this the idle launcher holds ~5.3G for the whole run; with
+# it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime
+# config, so they cannot affect the measured pass/fail signal.
+# * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command
+# below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends
+# so our -Xmx2G comes last and wins. Gluten offloads data to Velox
+# off-heap (capped at 2g via spark.memory.offHeap.size in the patched
+# DeltaSQLCommandTest), so the fork's heap need is modest. A larger
+# fork heap pushed the cgroup peak past the ~16G OOM threshold and the
+# kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G
+# keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is
+# analyzable.
+# `-u target/test-reports` enables ScalaTest's JUnit XML reporter so
+# every suite writes per-test results. Delta itself only configures
+# the console reporter (-oDF), so without this we'd have no machine-
+# readable results to gate on. The path is relative to the forked
+# test JVM's working dir (Test / baseDirectory = spark/), i.e.
+# delta/spark/target/test-reports/TEST-*.xml.
+#
+# We deliberately do NOT let an sbt non-zero exit (which fires on the
+# MANY expected Delta-on-Gluten failures) fail this step directly.
+# Instead the known-failures gate below decides pass/fail: the build
+# is green when the only failures are ones already recorded in the
+# baseline, and red on a genuine regression.
+set +e
+# --- hang watchdog ---------------------------------------------------
+# Shard 2 (and occasionally others) hangs indefinitely after a suite's
+# last test with no further output. ScalaTest's failAfter only wraps
+# individual test BODIES, so a wedge in suite teardown/afterAll -- or in
+# a non-interruptible native Velox/JNI call that ignores
+# Thread.interrupt() -- has no timeout and stalls until the 350-min job
+# limit with zero diagnostics. This watchdog dumps the forked test JVM's
+# threads (to the job log, and to a file for the artifact) once the test
+# output has been silent for too long, so the deadlock is diagnosable.
+SBT_LOG="/tmp/sbt-spark-test-shard-${SHARD_ID}.log"
+: > "$SBT_LOG"
+rm -f /tmp/sbt-done
+(
+ # CRITICAL: the step shell runs with `bash -eo pipefail`, which the
+ # subshell inherits. Without `set +e` here, ANY non-zero command --
+ # e.g. fork detection finding no match, or `kill`/`jps` returning
+ # non-zero -- silently kills this watchdog. That errexit kill (plus a
+ # /proc detection miss) once made the watchdog capture ZERO dumps. A
+ # diagnostic must never abort on a failed probe.
+ set +e +o pipefail
+ JSTACK="${JAVA_HOME}/bin/jstack"
+ JPS="${JAVA_HOME}/bin/jps"
+ silent_limit=900 # 15 min with no new test output => treat as hung
+ dumps=0
+ fork_pids() {
+ # The sbt test fork's main class is sbt.ForkMain. Prefer jps (reads
+ # the main class from hsperfdata, robust to sbt's @argfile launch);
+ # fall back to scanning /proc cmdline + @argfile.
+ "$JPS" -l 2>/dev/null | awk '/sbt\.ForkMain/ {print $1}'
+ local p cl arg
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] || continue
+ cl="$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)"
+ case "$cl" in *sbt.ForkMain*) echo "${p##*/}"; continue ;; esac
+ arg="$(printf '%s' "$cl" | tr ' ' '\n' | sed -n 's/^@//p' | head -1)"
+ [ -n "$arg" ] && [ -f "$arg" ] && grep -qa 'sbt\.ForkMain' "$arg" 2>/dev/null \
+ && echo "${p##*/}"
+ done
+ }
+ all_java_pids() {
+ "$JPS" -q 2>/dev/null
+ local p
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] && echo "${p##*/}"
+ done
+ }
+ echo "HANG WATCHDOG armed: dumps the test JVM after ${silent_limit}s of output silence"
+ hb=0
+ while [ ! -f /tmp/sbt-done ]; do
+ sleep 60
+ [ -f "$SBT_LOG" ] || continue
+ now=$(date +%s)
+ mtime=$(stat -c %Y "$SBT_LOG" 2>/dev/null || echo "$now")
+ silent=$(( now - mtime ))
+ # Per-minute memory profile: heap tuning proved the ~16G OOM peak is
+ # NATIVE-driven, so log which JVM (sbt launcher vs fork) actually grows
+ # toward it -- the last lines before a hang reveal the real hog to cut.
+ # Read /proc directly (no `ps` dependency in the minimal container).
+ memnow=$(awk '{printf "%.2fG",$1/1073741824}' /sys/fs/cgroup/memory.current 2>/dev/null)
+ jvmrss=""
+ for mp in $(all_java_pids 2>/dev/null | sort -un); do
+ r=$(awk '/^VmRSS:/{print $2}' "/proc/$mp/status" 2>/dev/null)
+ [ -n "$r" ] && jvmrss="$jvmrss $(( r / 1024 ))M(p$mp)"
+ done
+ echo "MEM cgroup=${memnow} JVMs=[${jvmrss# }]"
+ hb=$(( hb + 1 ))
+ # Heartbeat every ~5 min so we can SEE the watchdog is alive (and how
+ # long the test has been silent) without waiting for a hang.
+ [ $(( hb % 5 )) -eq 0 ] && echo "HANG WATCHDOG: alive; last test output ${silent}s ago"
+ if [ "$silent" -ge "$silent_limit" ] && [ "$dumps" -lt 3 ]; then
+ dumps=$(( dumps + 1 ))
+ pids="$(fork_pids | sort -un)"
+ # Safety net: if the fork JVM cannot be pinpointed, dump EVERY JVM.
+ [ -n "$pids" ] || pids="$(all_java_pids | sort -un)"
+ echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $pids))"
+ [ -n "$pids" ] || echo "HANG WATCHDOG: no java process found to dump"
+ for pid in $pids; do
+ # SIGQUIT makes the JVM print a full thread dump to its OWN stderr,
+ # which sbt relays into the test log via the SAME stream as test
+ # output -- so it lands in the job log even when a separately
+ # spawned jstack child's output would be buffered/lost. Also write
+ # jstack to a file for the per-shard artifact.
+ echo "----- SIGQUIT + jstack pid ${pid} -----"
+ kill -QUIT "$pid" 2>/dev/null || echo "HANG WATCHDOG: kill -QUIT failed for pid ${pid}"
+ timeout 120 "$JSTACK" -l "$pid" > "/tmp/threaddump-shard-${SHARD_ID}-${dumps}-${pid}.txt" 2>&1 \
+ || echo "HANG WATCHDOG: jstack failed/timed out for pid ${pid}"
+ done
+ echo "::endgroup::"
+ # The dump is now captured (job log via SIGQUIT + artifact via
+ # jstack file). A hung JVM otherwise stalls the whole shard until
+ # the 350-min job timeout AND keeps the job log frozen so the dump
+ # never becomes reachable. So KILL the wedged JVM(s): the suite
+ # fails fast (acceptable -- errors are expected; only an
+ # unrecoverable hang blocks CI), the job proceeds/ends, and the log
+ # + artifacts flush. Give SIGQUIT a moment to print first.
+ sleep 20
+ echo "HANG WATCHDOG: killing wedged JVM(s) to unblock the shard: $(printf '%s ' $pids)"
+ for pid in $pids; do kill -KILL "$pid" 2>/dev/null; done
Review Comment:
Good catch — you're reading it right, and this was a real gap. A hang+kill left the killed fork's running suite plus everything queued behind it in that JVM unrun and unreported, and since we ignore sbt's exit code the gate only judged the suites that reported, so the shard could go green. Fixed as you suggested: the watchdog now `touch`es a marker file when it kills a wedged fork, and after sbt returns the script fails the shard (`::error::` + `exit 1`) if the marker exists. A watchdog kill is an abnormal run, so it gets a red X. (73e5c90)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3617784035
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -94,6 +99,70 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Delta Spark UT, run via the reusable workflow so it reuses the native lib
+ # built above instead of duplicating the native build. Not gated on Delta-only
+ # paths: core/velox/substrait/cpp/shims changes can affect Delta query offload,
+ # so this runs on every trigger like the other spark-test jobs.
Review Comment:
The header comment here says the Delta Spark UT job “runs on every trigger”, but the actual logic below intentionally gates the reusable workflow per-PR (Delta-relevant paths or `run-delta-ci` label). This is a bit misleading for future maintainers; updating the comment to match the gating behavior will avoid confusion about when the Delta suite should run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3661118013
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,459 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib artifact it already built, so the expensive
+ # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a
+ # called workflow runs as part of the caller run), so the jobs below download it
+ # by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+ # Nightly full run against the latest default branch. The per-PR entry point
+ # (velox_backend_x86.yml) now runs the Delta suite only when a PR touches
+ # Delta-relevant paths (or carries the opt-in label), to save GHA minutes; this
+ # scheduled run keeps full coverage once a day so rarer regressions are still
+ # caught. It builds its own native lib (build-native-lib-centos-7 below) since
+ # there is no caller to provide one, and uses the workflow's default inputs.
+ schedule:
+ - cron: '0 5 * * *'
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. `spark_version` is the single source of truth for the
+ # Spark version: it drives the Gluten bundle profile (-Pspark-<v>), the bundle
+ # jar name, and Delta's -DsparkVersion, so the tests always run against a bundle
+ # built for the same Spark version (no separate value to keep in sync). Scala
+ # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from
+ # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a
+ # compatible delta_ref.
+ GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version || '4.1' }}
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version || '4.1' }}
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone runs (workflow_dispatch + nightly schedule) build the native lib
+ # here. When called by velox_backend_x86.yml the caller already built it and
+ # passes it as an input, so this job is skipped and the duplicate build avoided.
+ if: github.event_name == 'workflow_dispatch' || github.event_name == 'schedule'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Surface the inputs as step outputs. workflow_call / workflow_dispatch
+ # supply them (with defaults); the nightly `schedule` event supplies NONE,
+ # so fall back to the same defaults here. The boolean inputs are rendered
+ # as explicit 'true'/'false' strings (never empty) via the `&&/||` form so
+ # a schedule run resolves cleanly: update_baseline=false (enforce) and
+ # fail_on_fixed=true (so a now-passing baseline test turns the nightly red
+ # -- our signal that the committed baseline needs refreshing).
+ env:
+ DELTA_REF: ${{ inputs.delta_ref || 'v4.2.0' }}
+ SPARK_VERSION: ${{ inputs.spark_version || '4.1' }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism || '4' }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline && 'true' || 'false' }}
+ FAIL_ON_FIXED: ${{ github.event_name == 'schedule' && 'true' || (inputs.fail_on_fixed && 'true' || 'false') }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
+
+ - name: Download Gluten bundle jar
+ uses: actions/download-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: gluten-bundle
+
+ - name: Install minimal tools
+ run: |
+ set -euo pipefail
+ # apache/gluten:centos-9-jdk17 already has java-17, git, tar, a POSIX
+ # shell, and curl-minimal (which provides the `curl` command sbt's
+ # launcher needs). Install the rest of what Delta's build/sbt and the
+ # tests may need. We deliberately do NOT install the full `curl`
+ # package -- it conflicts with the pre-installed curl-minimal.
+ yum install -y java-17-openjdk-devel which findutils gzip python3
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ git --version
+ curl --version | head -n 1
+
+ - name: Cache sbt / Ivy / Coursier
+ uses: actions/cache@v4
+ with:
+ path: |
+ /root/.sbt
+ /root/.ivy2
+ /root/.cache/coursier
+ # Intentionally NOT keyed by ${{ matrix.shard }} -- all shards share
+ # the same dependency tree, so a single shared cache (with parallel
+ # save races resolved by GH on a first-write-wins basis) gives the
+ # best storage / hit-rate tradeoff.
+ key: delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-${{ env.DELTA_SCALA_VERSION }}
+ restore-keys: |
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-
Review Comment:
The sbt/Ivy/Coursier cache key includes `${{ steps.resolve.outputs.delta_ref }}`. Since `delta_ref` is a workflow input that may be a branch name (often containing `/`), this can produce an invalid cache key for `actions/cache` and fail the job on otherwise-valid manual runs.
Sanitize `delta_ref` (and keep restore-keys consistent) before using it in the cache key, e.g. replace `/` with `-`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5339300614
@felipepessoto could you please move the cache to apache stash based? GHA only have 10GB avaiable, and it's occupying too many space, thus the cache on main branch is evicted and introducing cold build
https://github.com/apache/gluten/actions/caches
<img width="1415" height="253" alt="image" src="https://github.com/user-attachments/assets/e98fdafd-48d8-4e3e-96b9-af40dbd904e8" />
Cc: @philo-he
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3502474493
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,484 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline file exists but is empty (not yet bootstrapped) the mode
+ automatically degrades to ``seed`` so the first run is never spuriously red.
+ A *missing* ``--known-failures`` file is treated as a configuration error
+ (the gate fails) so a mis-referenced path can't silently pass.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard).
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import glob
+import os
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped) sets of (suite, test) tuples. A test is
+ 'failed' if its <testcase> has a <failure> or <error> child, 'skipped' if
+ it has a <skipped> child, otherwise 'passed'. Suite-level aborts (a
+ <testsuite> reporting errors/failures with no failing <testcase>) are
+ recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = (suite, name)
+ tags = _child_local_tags(tc)
+ if "failure" in tags or "error" in tags:
+ failed.add(key)
+ suite_has_failing_tc = True
+ elif "skipped" in tags:
+ skipped.add(key)
+ else:
+ passed.add(key)
+
+ # Suite-level abort: counters say something failed but no testcase
+ # carried the failure (the suite blew up in beforeAll/constructor).
+ # Record a
+ # synthetic entry so the regression is visible.
+ try:
+ errors = int(ts.get("errors", "0") or "0")
+ failures = int(ts.get("failures", "0") or "0")
+ except ValueError:
+ errors = failures = 0
+ if (errors + failures) > 0 and not suite_has_failing_tc:
+ failed.add((suite_name, SUITE_ABORTED))
+
+ if not parsed_any:
+ eprint(
+ "WARNING: no JUnit <testsuite> elements found under {}".format(reports_dir)
+ )
+
+ # A test can't be both passed and failed; failure wins. Skipped only counts
+ # if the test was not otherwise seen (e.g. retried).
+ passed -= failed
+ skipped -= failed
+ skipped -= passed
+ return passed, failed, skipped
+
+
+# --------------------------------------------------------------------------- #
+# Reporting helpers
+# --------------------------------------------------------------------------- #
+def _summary_sink():
+ """Return a writer that mirrors to GITHUB_STEP_SUMMARY when available."""
+ path = os.environ.get("GITHUB_STEP_SUMMARY")
+ handle = open(path, "a", encoding="utf-8") if path else None
+
+ def write(line=""):
+ print(line)
+ if handle:
+ handle.write(line + "\n")
+
+ return write, handle
+
+
+def _print_block(write, title, entries, limit=50):
+ write("")
+ write("### {} ({})".format(title, len(entries)))
+ if not entries:
+ return
+ write("")
+ write("```")
+ for i, (suite, test) in enumerate(sorted(entries)):
+ if i >= limit:
+ write("... and {} more".format(len(entries) - limit))
+ break
+ write(format_entry(suite, test))
+ write("```")
+
+
+# --------------------------------------------------------------------------- #
+# Modes
+# --------------------------------------------------------------------------- #
+def run_enforce(args):
+ # In enforce mode a missing baseline file would make load_entries() return an
+ # empty set, silently degrading to seed mode and passing the gate without
+ # enforcing anything. Treat a missing path as a configuration error; an
+ # existing-but-empty file is still allowed (it legitimately seeds).
+ if args.mode == "enforce" and (
+ not args.known_failures or not os.path.exists(args.known_failures)
+ ):
+ eprint(
+ "ERROR: --known-failures '{}' does not exist. In enforce mode the "
+ "baseline file must exist (an existing-but-empty file is allowed and "
+ "triggers seed mode). Refusing to silently pass.".format(
+ args.known_failures
+ )
+ )
+ return 2
+ baseline = load_entries(args.known_failures)
+ passed, failed, skipped = parse_reports(args.reports_dir)
+
+ # Always emit this shard's artifacts for the aggregation job.
+ if args.failures_out:
+ write_entries(args.failures_out, failed)
+ if args.ran_out:
+ write_entries(args.ran_out, passed | failed)
+
+ write, handle = _summary_sink()
+ try:
+ seeding = args.mode == "seed" or not baseline
+ if seeding and args.mode != "seed":
+ write(
+ "> NOTE: baseline `{}` is empty -- running in SEED mode "
+ "(no failures will be enforced). Bootstrap the baseline from "
+ "the aggregated artifact, commit it, then enforcement begins.".format(
+ args.known_failures
+ )
+ )
+
+ write(
+ "## Delta-on-Gluten test gate -- shard {}".format(
+ os.environ.get("SHARD_ID", "?")
+ )
+ )
+ write("")
+ write("| Category | Count |")
+ write("|---|---:|")
+ write("| Ran (pass+fail) | {} |".format(len(passed) + len(failed)))
+ write("| Passed | {} |".format(len(passed)))
+ write("| Failed | {} |".format(len(failed)))
+ write("| Skipped | {} |".format(len(skipped)))
+ write("| Baseline (known failures) | {} |".format(len(baseline)))
+
+ if seeding:
+ write("")
+ write(
+ "Seed mode: recorded {} failing test(s) for this shard. "
+ "Nothing enforced.".format(len(failed))
+ )
+ return 0
+
+ regressions = failed - baseline
+ fixed = baseline & passed
+ expected = failed & baseline
+
+ write("")
+ write("| Gate result | Count |")
+ write("|---|---:|")
+ write("| Expected failures (in baseline) | {} |".format(len(expected)))
+ write("| **Regressions (new failures)** | {} |".format(len(regressions)))
+ write("| Now-passing (remove from baseline) | {} |".format(len(fixed)))
+
+ _print_block(
+ write, "Regressions -- new failures NOT in the baseline", regressions
+ )
+ if regressions:
+ write("")
+ write(
+ "These tests were not previously known to fail. Either fix "
+ "the regression, or (if it is a genuinely new expected "
+ "failure) add the lines above to `known-failures.txt`."
+ )
+
+ if args.fail_on_fixed:
+ _print_block(
+ write, "Now-passing -- delete these lines from the baseline", fixed
+ )
+
+ exit_code = 0
+ if regressions:
+ for suite, test in sorted(regressions):
+ eprint("::error::REGRESSION {}".format(format_entry(suite, test)))
+ exit_code = 1
+ if args.fail_on_fixed and fixed:
+ for suite, test in sorted(fixed):
+ eprint(
+ "::error::NOW-PASSING (remove from baseline) {}".format(
+ format_entry(suite, test)
+ )
+ )
+ exit_code = 1
+
+ if exit_code == 0:
+ write("")
+ write("All failures are expected (in the baseline). Gate passed.")
+ return exit_code
+ finally:
+ if handle:
+ handle.close()
+
+
+def run_aggregate(args):
+ failure_files = sorted(
+ glob.glob(os.path.join(args.inputs_dir, "**", "failures-*.txt"), recursive=True)
+ )
+ ran_files = sorted(
+ glob.glob(os.path.join(args.inputs_dir, "**", "ran-*.txt"), recursive=True)
+ )
+
Review Comment:
Done in 6fe545120. Aggregate mode now exits 2 before writing --baseline-out when no per-shard failures-*.txt / ran-*.txt files are found. Since the gate-list download is continue-on-error and the aggregate job runs with if: always() , missing artifacts would otherwise produce an empty baseline that could be committed and wipe known-failures.txt ; that path is now a hard error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-4967373114
@felipepessoto could you please point me to the tests that you think might be failing due to TimestampNTZ? I'd like to take a closer look. It appears that the tests in the log you shared here https://github.com/apache/gluten/issues/11622#issuecomment-4965875346 finished without failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-4971044502
@felipepessoto I see. I will take a look, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3549295317
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,692 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. `spark_version` is the single source of truth for the
+ # Spark version: it drives the Gluten bundle profile (-Pspark-<v>), the bundle
+ # jar name, and Delta's -DsparkVersion, so the tests always run against a bundle
+ # built for the same Spark version (no separate value to keep in sync). Scala
+ # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from
+ # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a
+ # compatible delta_ref.
+ GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version }}
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version }}
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
Review Comment:
The number of shards shouldn't impact much the usage, there is not much waste for each shard initialization. If we had a single VM it would take 9h+ to run all the tests. It is more 4 x 2h20m vs 1 x 9h.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-4928578615
Heads-up for reviewers: since the last review round I added **flaky-test handling** to the gate (it wasn't in the original description), and just reworked it to match by **error signature** rather than test name. Summarising so nobody has to reverse-engineer it from the diffs.
**The problem.** A handful of Delta MERGE-with-deletion-vector tests fail *non-deterministically*: the same bundle passes them on one run and fails them on the next. They can't go in `known-failures.txt` — baselining them reds the gate when they pass (`fail_on_fixed`), and leaving them out reds it when they fail. The root cause is a native bug: the DV bitmap aggregator aborts on an invalid `Long.MAX_VALUE` row index during a DV-writing MERGE (`RoaringBitmapArray.cpp` `addSafe`, `value <= kMaxRepresentableValue`). It's intermittent and lands on a **different `*DVs*Suite` MERGE test each run**.
**What the gate now does.** Two ways to quarantine a flake (a quarantined failure is neutral — never a regression, never now-passing, and excluded from the regenerated baseline):
1. `flaky-tests.txt` — by **test name** (`<suite-glob>#<test>`), for ordinary flakes.
2. `flaky-error-patterns.txt` — by **error signature**: a regex matched against the failed test's JUnit `<failure>`/`<error>` text. Seeded with the DV-bitmap signature:
```
Delta RoaringBitmapArray row index \d+ exceeds max representable value
```
Because this intermittent bug hits a different test each run, matching by name was whack-a-mole (it had grown to 6 entries). Matching the signature ignores it on **whichever** test it lands, so I removed those 6 name entries. It's also *more precise* than a name glob: a **different** real failure in the same DV suite is still caught as a regression, since only failures carrying the signature are ignored.
This is an **interim** measure — it hides a real native bug from CI. The pattern references the tracking issue and should be removed once the row-index bug is fixed in the native backend.
Verified against a real failing shard report: the DV failure is reported as a regression without the patterns file, and quarantined (gate green) with it. See `.github/workflows/util/delta-spark-ut/README.md` → "Flaky tests" for details. Relevant commits: `0a3759f77` (initial flaky quarantine) and `cd024df86` (signature-based).
I've refreshed the PR description to cover this (plus the Arrow-jar removal and the script extraction below).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3549772974
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,692 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. `spark_version` is the single source of truth for the
+ # Spark version: it drives the Gluten bundle profile (-Pspark-<v>), the bundle
+ # jar name, and Delta's -DsparkVersion, so the tests always run against a bundle
+ # built for the same Spark version (no separate value to keep in sync). Scala
+ # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from
+ # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a
+ # compatible delta_ref.
+ GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version }}
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version }}
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Every input has a default (workflow_call + workflow_dispatch), so they
+ # are always set; just surface them as step outputs for the steps below.
+ env:
+ DELTA_REF: ${{ inputs.delta_ref }}
+ SPARK_VERSION: ${{ inputs.spark_version }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline }}
+ FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
+
+ - name: Download Gluten bundle jar
+ uses: actions/download-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: gluten-bundle
+
+ - name: Install minimal tools
+ run: |
+ set -euo pipefail
+ # apache/gluten:centos-9-jdk17 already has java-17, git, tar, a POSIX
+ # shell, and curl-minimal (which provides the `curl` command sbt's
+ # launcher needs). Install the rest of what Delta's build/sbt and the
+ # tests may need. We deliberately do NOT install the full `curl`
+ # package -- it conflicts with the pre-installed curl-minimal.
+ yum install -y java-17-openjdk-devel which findutils gzip python3
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ git --version
+ curl --version | head -n 1
+
+ - name: Cache sbt / Ivy / Coursier
+ uses: actions/cache@v4
+ with:
+ path: |
+ /root/.sbt
+ /root/.ivy2
+ /root/.cache/coursier
+ # Intentionally NOT keyed by ${{ matrix.shard }} -- all shards share
+ # the same dependency tree, so a single shared cache (with parallel
+ # save races resolved by GH on a first-write-wins basis) gives the
+ # best storage / hit-rate tradeoff.
+ key: delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-${{ env.DELTA_SCALA_VERSION }}
+ restore-keys: |
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-
+
+ - name: Clone and patch Delta
+ run: |
+ set -euo pipefail
+ # `2>/dev/null ... || true` keeps a no-match `ls` from aborting the step
+ # under `set -o pipefail`, so the explicit check below emits a clear
+ # error instead of a generic "cannot access".
+ GLUTEN_JAR=$(ls "$GITHUB_WORKSPACE"/gluten-bundle/gluten-velox-bundle-spark*_*-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$GLUTEN_JAR" ] || [ ! -f "$GLUTEN_JAR" ]; then
+ echo "ERROR: No Gluten bundle jar found under $GITHUB_WORKSPACE/gluten-bundle/" >&2
+ ls -la "$GITHUB_WORKSPACE/gluten-bundle/" || true
+ exit 1
+ fi
+ echo "Using Gluten bundle: $GLUTEN_JAR"
+ bash "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/setup-delta.sh" \
+ "${{ steps.resolve.outputs.delta_ref }}" \
+ "$GITHUB_WORKSPACE/delta" \
+ "$GLUTEN_JAR" \
+ "$GITHUB_WORKSPACE"
+
+ - name: Run Delta spark module tests (shard ${{ matrix.shard }} / ${{ env.DELTA_NUM_SHARDS }})
+ env:
+ NUM_SHARDS: ${{ env.DELTA_NUM_SHARDS }}
+ TEST_PARALLELISM_COUNT: ${{ steps.resolve.outputs.test_parallelism }}
+ # Required by Delta to enable testing-only code paths
+ # (see delta build.sbt: "Test / envVars += DELTA_TESTING -> 1").
+ DELTA_TESTING: '1'
+ # JDK 17 + Gluten/Arrow/Netty requires extra --add-opens and the
+ # `io.netty.tryReflectionSetAccessible` system property; otherwise
+ # the forked test JVM fails with
+ # java.lang.UnsupportedOperationException: sun.misc.Unsafe or
+ # java.nio.DirectByteBuffer.<init>(long, int) not available
+ # as soon as Gluten's bundled Arrow allocator initializes Netty
+ # direct buffers. Delta's own `Test / javaOptions` (see
+ # project/CrossSparkVersions.scala `java17TestSettings`) sets the
+ # base add-opens but NOT the Netty property -- Delta's own tests
+ # don't load Arrow/Netty buffers in a way that triggers it.
+ #
+ # Use JAVA_TOOL_OPTIONS so the flags propagate to BOTH the sbt
+ # launcher JVM and the forked test JVM (sbt forks tests and the
+ # child inherits the parent's env). The set below mirrors
+ # `extraJavaTestArgs` from Gluten's own root pom.xml (the
+ # canonical Gluten test JVM flag set).
+ #
+ # NOTE: we deliberately do NOT put `-Xmx` here. JAVA_TOOL_OPTIONS
+ # is processed BEFORE the JVM command line, so Delta's explicit
+ # `-Xmx1024m` (set in build.sbt `Test / javaOptions`) would still
+ # win (last `-Xmx` wins). The forked-test-JVM heap is bumped via
+ # an sbt `set spark / Test / javaOptions ++= ...` command below,
+ # which APPENDS to Delta's own seq -- so our `-Xmx` lands AFTER
+ # `-Xmx1024m` and wins.
+ JAVA_TOOL_OPTIONS: >-
+ -XX:+IgnoreUnrecognizedVMOptions
+ --add-opens=java.base/java.lang=ALL-UNNAMED
+ --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
+ --add-opens=java.base/java.lang.reflect=ALL-UNNAMED
+ --add-opens=java.base/java.io=ALL-UNNAMED
+ --add-opens=java.base/java.net=ALL-UNNAMED
+ --add-opens=java.base/java.nio=ALL-UNNAMED
+ --add-opens=java.base/java.util=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
+ --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.cs=ALL-UNNAMED
+ --add-opens=java.base/sun.security.action=ALL-UNNAMED
+ --add-opens=java.base/sun.util.calendar=ALL-UNNAMED
+ -Djdk.reflect.useDirectMethodHandle=false
+ -Dio.netty.tryReflectionSetAccessible=true
+ -Dfile.encoding=UTF-8
Review Comment:
Added util/delta-spark-ut/java-test-args.sh with those JVM flags, run-delta-tests.sh sources it and a developer can source the same file before sbt spark/test to get identical settings (README has a "Running the suite locally" note). I kept it as an exported JAVA_TOOL_OPTIONS rather than sbt .jvmopts / .sbtopts because the flags need to reach the forked test JVM (which inherits the env), whereas .jvmopts only configures the sbt launcher.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3669855867
##########
.github/workflows/util/delta-spark-ut/run-delta-tests.sh:
##########
@@ -0,0 +1,275 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Runs the Delta `spark` module tests for one shard under the Gluten bundle:
+# arms a hang watchdog (thread-dumps + kills a wedged fork), invokes sbt
+# spark/test with the tuned JVM/heap flags, prints cgroup memory forensics, and
+# then gates the results against the baseline (compare-test-results.py). Extracted
+# from delta_spark_ut.yml so the workflow step stays readable.
+#
+# Driven by environment (set by the workflow step / job):
+# SHARD_ID - this shard's id (matrix.shard)
+# SPARK_VERSION - Delta -DsparkVersion value
+# UPDATE_BASELINE - 'true' -> gate seed mode; else enforce
+# FAIL_ON_FIXED - passed through to the gate
+# DELTA_SCALA_VERSION, NUM_SHARDS, TEST_PARALLELISM_COUNT, DELTA_TESTING
+# - test env (see the workflow step's `env:` block)
+# GITHUB_WORKSPACE - repo root (holds the Delta clone + util scripts)
+#
+# JAVA_TOOL_OPTIONS is set by sourcing java-test-args.sh (below), not the caller.
+
+set -euo pipefail
+export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+export PATH=$JAVA_HOME/bin:$PATH
+# Gluten/JDK17 test JVM flags (--add-opens + Netty property), shared with local
+# dev runs. Sets JAVA_TOOL_OPTIONS so it reaches the sbt launcher + forked JVMs.
+# shellcheck source=./java-test-args.sh
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/java-test-args.sh"
+cd "$GITHUB_WORKSPACE/delta"
+chmod +x build/sbt
+# Only run the unified `spark` sbt project, NOT `sparkGroup/test` --
+# `sparkGroup` aggregates many other projects (sparkV2, contribs,
+# sharing, connect*, ...) that are out of scope for this pipeline.
+#
+# JVM heap layout -- two memory consumers on the ~16G runner:
+# * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to
+# return idle memory during the (long) test phase via G1 periodic GC
+# (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the
+# busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent
+# forces each periodic GC to a full STW collection) that uncommits to a
+# tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low
+# -Xms512m floor.
+# Without this the idle launcher holds ~5.3G for the whole run; with
+# it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime
+# config, so they cannot affect the measured pass/fail signal.
+# * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command
+# below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends
+# so our -Xmx2G comes last and wins. Gluten offloads data to Velox
+# off-heap (capped at 2g via spark.memory.offHeap.size in the patched
+# DeltaSQLCommandTest), so the fork's heap need is modest. A larger
+# fork heap pushed the cgroup peak past the ~16G OOM threshold and the
+# kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G
+# keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is
+# analyzable.
+# `-u target/test-reports` enables ScalaTest's JUnit XML reporter so
+# every suite writes per-test results. Delta itself only configures
+# the console reporter (-oDF), so without this we'd have no machine-
+# readable results to gate on. The path is relative to the forked
+# test JVM's working dir (Test / baseDirectory = spark/), i.e.
+# delta/spark/target/test-reports/TEST-*.xml.
+#
+# We deliberately do NOT let an sbt non-zero exit (which fires on the
+# MANY expected Delta-on-Gluten failures) fail this step directly.
+# Instead the known-failures gate below decides pass/fail: the build
+# is green when the only failures are ones already recorded in the
+# baseline, and red on a genuine regression.
+set +e
+# --- hang watchdog ---------------------------------------------------
+# Shard 2 (and occasionally others) hangs indefinitely after a suite's
+# last test with no further output. ScalaTest's failAfter only wraps
+# individual test BODIES, so a wedge in suite teardown/afterAll -- or in
+# a non-interruptible native Velox/JNI call that ignores
+# Thread.interrupt() -- has no timeout and stalls until the 350-min job
+# limit with zero diagnostics. This watchdog dumps the forked test JVM's
+# threads (to the job log, and to a file for the artifact) once the test
+# output has been silent for too long, so the deadlock is diagnosable.
+SBT_LOG="/tmp/sbt-spark-test-shard-${SHARD_ID}.log"
+# Marker the watchdog touches when it KILLS a wedged test fork. The killed fork's
+# running suite plus every suite queued behind it never run and never write a
+# report, and since we ignore sbt's exit code the gate would only judge the
+# suites that DID report -- so the main flow fails the shard when this exists.
+WATCHDOG_KILL_MARKER="/tmp/sbt-watchdog-killed-shard-${SHARD_ID}"
+: > "$SBT_LOG"
+rm -f /tmp/sbt-done "$WATCHDOG_KILL_MARKER"
Review Comment:
Valid — fixed in 15f06bad5. You're right that it was inconsistent: the script already scopes `SBT_LOG` and `WATCHDOG_KILL_MARKER` by `SHARD_ID`, and the done marker was the odd one out.
It's now `/tmp/sbt-done-shard-${SHARD_ID}` via a named `SBT_DONE_MARKER`, covering all three sites you flagged (the `rm -f`, the watchdog's wait loop, and the `touch` after sbt returns).
I reproduced the arm/disarm handshake with a fast shard running alongside a slow one to confirm the impact was real rather than cosmetic:
| done marker | slow shard's watchdog |
|---|---|
| `/tmp/sbt-done` (global) | disarmed after **3** ticks — hang detection lost |
| `/tmp/sbt-done-shard-N` | stayed armed all **10** ticks — correct |
In CI each shard gets its own container, so this was latent there; it matters for parallel local runs, which is exactly the case you described.
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -101,6 +106,80 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR
+ # touches high-signal Delta paths -- the Delta integration code
+ # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own
+ # files -- or carries the `run-delta-ci` opt-in label. Changes to general
+ # Velox/core/native code can also affect Delta offload but are touched
+ # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml
+ # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage
+ # down. NOTE: the label is read from the event that triggered this run, so add
+ # it before/with a push; labeling an already-finished run needs a new push.
+ delta-changes:
+ runs-on: ubuntu-22.04
+ outputs:
+ run_delta: ${{ steps.filter.outputs.run_delta }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ - name: Detect Delta-relevant changes / opt-in label
+ id: filter
+ env:
+ HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
Review Comment:
Checked again, but I don't think this applies — no change. (Same underlying concern as [#discussion_r3659868584](https://github.com/apache/gluten/pull/12388#discussion_r3659868584) and [#discussion_r3660828861](https://github.com/apache/gluten/pull/12388#discussion_r3660828861).)
Two independent reasons:
1. **The premise can't occur here.** Parsing this workflow's triggers yields exactly `['pull_request']` — there is no `schedule`, `push` or `workflow_dispatch` entry — so `github.event.pull_request` is always present. (The reusable `delta_spark_ut.yml` has the other triggers, but it doesn't evaluate this expression.)
2. **Even on a non-PR event it wouldn't error.** Actions coerces a missing context to a falsy empty value rather than failing evaluation — per the [expressions reference](https://docs.github.com/en/actions/reference/workflows-and-actions/expressions), `Null` casts to `''` and falsy values coerce to `false`. Concretely, [`n8n-io/n8n`'s `release-storybook.yml`](https://github.com/n8n-io/n8n/blob/master/.github/workflows/release-storybook.yml) runs on a nightly `schedule` and evaluates `!contains(github.event.pull_request.labels.*.name, 'community')` with no event guard at all — it would break every night if this failed evaluation. The pattern is paired with `schedule:` in well over a thousand public workflows.
So an `event_name == 'pull_request'` guard would be unconditionally true here and would only add noise. The shell-level fail-open on missing SHAs stays as belt-and-suspenders.
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -101,6 +106,80 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR
+ # touches high-signal Delta paths -- the Delta integration code
+ # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own
+ # files -- or carries the `run-delta-ci` opt-in label. Changes to general
+ # Velox/core/native code can also affect Delta offload but are touched
+ # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml
+ # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage
+ # down. NOTE: the label is read from the event that triggered this run, so add
+ # it before/with a push; labeling an already-finished run needs a new push.
+ delta-changes:
+ runs-on: ubuntu-22.04
+ outputs:
+ run_delta: ${{ steps.filter.outputs.run_delta }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ - name: Detect Delta-relevant changes / opt-in label
+ id: filter
+ env:
+ HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+ run: |
+ set -euo pipefail
+ # Opt-in label forces the suite even with no Delta-relevant path change.
+ if [ "$HAS_LABEL" = "true" ]; then
+ echo "run-delta-ci label present -> running Delta suite"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
+ fi
+ # Fail open if we can't determine the PR range (e.g. a non-PR trigger):
+ # never silently skip coverage.
+ if [ -z "${BASE_SHA:-}" ] || [ -z "${HEAD_SHA:-}" ]; then
+ echo "no PR base/head sha -> running Delta suite (fail-open)"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
+ fi
+ BASE=$(git merge-base "$BASE_SHA" "$HEAD_SHA" 2>/dev/null || echo "$BASE_SHA")
+ echo "diff base=$BASE head=$HEAD_SHA"
+ # Fail open if the diff itself can't be computed (missing objects after a
+ # force-push race, an unfetched fork head, ...). Piping straight into
+ # `grep -q` inside an `if` would hide that: git's failure leaves grep with
+ # empty input, so the pipeline exits non-zero exactly as it does for "no
+ # match" -- and `set -e`/`pipefail` can't help, since a tested command is
+ # allowed to fail. Capture the diff first so the two cases stay distinct.
+ if ! CHANGED=$(git diff --name-only "$BASE" "$HEAD_SHA"); then
Review Comment:
Good hypothesis, but it's empirically not what happens — no change.
This PR is itself a fork PR (`felipepessoto/gluten` → `apache/gluten`), so it exercises exactly the path in question. Across every run of the gate so far, `git diff` succeeded and the gate reached a real decision — it never hit the fail-open branch:
```
diff base=405e2b67e... head=3e5d4947b...
Delta-relevant paths changed -> running Delta suite
```
(4/4 runs printed `Delta-relevant paths changed`; none printed `git diff failed -> ... (fail-open)`.)
The reason it works is that the job checks out `refs/pull/<n>/merge` with `fetch-depth: 0`, and that merge commit's **second parent is the fork head commit** — so `github.event.pull_request.head.sha` is present in the local object store, along with the base. Switching to `github.sha` would actually diff the *merge* commit, which also pulls in base-branch changes since the fork point and would make the gate fire more often, not less.
The fail-open branch stays as the safety net for genuine diff failures, which is what it was added for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5121160005
@zhouyuan, please check these two comments:
https://github.com/apache/gluten/pull/12388#issuecomment-4989583126
https://github.com/apache/gluten/pull/12388#issuecomment-5015174080
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5341041673
@zhouyuan Thanks for pointing this out. I opened #12820 to move all Delta Spark UT caches (ccache, Maven, and sbt/Ivy/Coursier) from `actions/cache` to Apache Stash. The [full Delta workflow](https://github.com/apache/gluten/actions/runs/32232899547) passed: ccache restored successfully from the main-branch stash, and the Maven and sbt Stash artifacts were uploaded successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3659868584
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -101,6 +106,70 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR
+ # touches high-signal Delta paths -- the Delta integration code
+ # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own
+ # files -- or carries the `run-delta-ci` opt-in label. Changes to general
+ # Velox/core/native code can also affect Delta offload but are touched
+ # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml
+ # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage
+ # down. NOTE: the label is read from the event that triggered this run, so add
+ # it before/with a push; labeling an already-finished run needs a new push.
+ delta-changes:
+ runs-on: ubuntu-22.04
+ outputs:
+ run_delta: ${{ steps.filter.outputs.run_delta }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ - name: Detect Delta-relevant changes / opt-in label
+ id: filter
+ env:
+ HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
Review Comment:
These expressions dereference `github.event.pull_request.*` unconditionally, but the step script explicitly anticipates non-PR triggers. On non-PR events, this can fail at workflow evaluation time before the shell `fail-open` logic runs. Guard the expressions (e.g., condition on `github.event_name == 'pull_request'`) and default to empty strings/false when not a PR.
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,208 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# init + shallow fetch resolves a tag, branch OR commit SHA in a single path
+# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the
+# destructive `rm -rf "$DELTA_DIR"` it required. `--` terminates options so a
+# DELTA_REF starting with `-` can't be misread as a git flag (this script is
+# workflow_dispatch-runnable with a user-supplied ref).
+git init -q "$DELTA_DIR"
+git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git
+git -C "$DELTA_DIR" fetch -q --depth 1 origin -- "$DELTA_REF"
+git -C "$DELTA_DIR" checkout -q FETCH_HEAD
+git -C "$DELTA_DIR" --no-pager log -1 --oneline
+echo "::endgroup::"
+
+echo "::group::Injecting Gluten bundle jar onto the spark project's TEST classpath"
+# The Gluten bundle jar must be on the spark project's TEST runtime classpath
+# (so DeltaSQLCommandTest can load org.apache.gluten.GlutenPlugin by name) but
+# NOT on the COMPILE classpath of `sparkV1`, which is the project that holds
+# Delta's main sources. The bundle's transitive contents include extra symbols
+# under `org.apache.spark.sql` that collide with Delta's main sources -- e.g.
+# MergeOutputGeneration.scala imports both `org.apache.spark.sql._` and
+# `org.apache.spark.sql.delta.ClassicColumnConversions._`, and would then fail
+# with `reference to expression is ambiguous`.
+#
+# sbt auto-scans `<baseDirectory>/lib` via `unmanagedBase`. Two relevant
+# projects in Delta v4.2.0 have a `lib/` baseDirectory:
+# - sparkV1: `project in file("spark")` -> spark/lib
+# - spark : `project in file("spark-unified")` -> spark-unified/lib
+# unmanagedJars are project-scoped (NOT inherited by dependents), so dropping
+# the bundle into spark-unified/lib/ adds it to the unified `spark` project's
+# Compile *and* Test classpaths -- but NOT to sparkV1's. That's exactly what
+# we want:
+# * sparkV1/Compile sees ONLY Delta's regular deps -> Delta main compiles.
+# * spark/Test/fullClasspath sees the bundle -> tests load GlutenPlugin.
+# (Verified empirically: with bundle only in spark-unified/lib/, sbt's
+# `show sparkV1/Compile/dependencyClasspath` excludes the bundle and
+# `show spark/Test/fullClasspath` includes it.)
+#
+# We deliberately do NOT also drop the bundle into spark/lib/, which is what
+# caused the previous compile failure: spark/lib/ is sparkV1's unmanagedBase,
+# and putting the bundle there would re-introduce the ambiguity errors.
+SPARK_UNIFIED_LIB="$DELTA_DIR/spark-unified/lib"
+mkdir -p "$SPARK_UNIFIED_LIB"
+cp "$GLUTEN_BUNDLE_JAR" "$SPARK_UNIFIED_LIB/gluten-velox-bundle.jar"
+ls -lh "$SPARK_UNIFIED_LIB"
+echo "::endgroup::"
+
+echo "::group::Patching DeltaSQLCommandTest to enable Gluten plugin"
+TARGET="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$TARGET" ]; then
+ echo "Expected file not found in Delta clone: $TARGET" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
+ exit 1
+fi
+cp "$PATCH_SOURCE" "$TARGET"
+echo "Patched $TARGET"
+echo "--- diff vs. upstream ---"
+git -C "$DELTA_DIR" --no-pager diff -- "spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala" || true
+echo "::endgroup::"
+
+# Delta's tests collect file-source scans by matching the concrete
+# `FileSourceScanExec` case class; Gluten offloads the scan to
+# DeltaScanTransformer, a `FileSourceScanLike` sibling, so those matches miss
+# (`scala.MatchError: List()`, empty partition filters, broken column-pruning /
+# scan-metric checks across many suites). delta-io/delta#7104 and #7105 widen the
+# matches to the shared `FileSourceScanLike` interface that both the vanilla and
+# Gluten scans implement (behavior-preserving for vanilla). Both are merged
+# upstream but land after the pinned DELTA_REF (v4.2.0), so apply them here; once
+# DELTA_REF includes a commit its cherry-pick is a clean no-op and the call can go.
+#
+# Depth-2 fetch brings each fix commit and its parent, which cherry-pick needs to
+# diff against (a depth-1 fetch grafts the parent away); `-n` stages the change
+# without requiring a committer identity.
+cherry_pick_delta_fix() {
+ local sha="$1" pr="$2"
+ echo "Cherry-picking delta-io/delta${pr}"
+ git -C "$DELTA_DIR" fetch --quiet --depth 2 origin "$sha"
+ git -C "$DELTA_DIR" cherry-pick -n "$sha"
+}
+
+echo "::group::Cherry-picking upstream Delta FileSourceScanLike test fixes"
+cherry_pick_delta_fix 46bd45d57eadd7e528002a0ae7bd36ce5a456eca "#7104 (ScanReportHelper.collectScans)"
+cherry_pick_delta_fix 959e00e15f41f56afc1c9bb95d160c55c6dc7068 "#7105 (9 more test suites)"
Review Comment:
Despite the comment claiming this becomes a 'clean no-op' when `DELTA_REF` already contains the fix, `git cherry-pick` typically fails (empty change / already applied) and can leave the repo in a cherry-pick state, breaking the workflow for newer Delta refs. Add an explicit 'already contained' check (e.g., `merge-base --is-ancestor`) to skip, or handle empty cherry-picks by aborting/skipping safely.
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,459 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib artifact it already built, so the expensive
+ # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a
+ # called workflow runs as part of the caller run), so the jobs below download it
+ # by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+ # Nightly full run against the latest default branch. The per-PR entry point
+ # (velox_backend_x86.yml) now runs the Delta suite only when a PR touches
+ # Delta-relevant paths (or carries the opt-in label), to save GHA minutes; this
+ # scheduled run keeps full coverage once a day so rarer regressions are still
+ # caught. It builds its own native lib (build-native-lib-centos-7 below) since
+ # there is no caller to provide one, and uses the workflow's default inputs.
+ schedule:
+ - cron: '0 5 * * *'
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
Review Comment:
Setting `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true` relaxes GitHub Actions Node runtime restrictions and can re-enable deprecated/insecure Node versions for actions. If not strictly required, remove it; otherwise, prefer updating/pinning actions to versions that run on supported Node runtimes so this flag isn't needed.
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,459 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib artifact it already built, so the expensive
+ # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a
+ # called workflow runs as part of the caller run), so the jobs below download it
+ # by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+ # Nightly full run against the latest default branch. The per-PR entry point
+ # (velox_backend_x86.yml) now runs the Delta suite only when a PR touches
+ # Delta-relevant paths (or carries the opt-in label), to save GHA minutes; this
+ # scheduled run keeps full coverage once a day so rarer regressions are still
+ # caught. It builds its own native lib (build-native-lib-centos-7 below) since
+ # there is no caller to provide one, and uses the workflow's default inputs.
+ schedule:
+ - cron: '0 5 * * *'
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. `spark_version` is the single source of truth for the
+ # Spark version: it drives the Gluten bundle profile (-Pspark-<v>), the bundle
+ # jar name, and Delta's -DsparkVersion, so the tests always run against a bundle
+ # built for the same Spark version (no separate value to keep in sync). Scala
+ # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from
+ # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a
+ # compatible delta_ref.
+ GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version || '4.1' }}
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version || '4.1' }}
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone runs (workflow_dispatch + nightly schedule) build the native lib
+ # here. When called by velox_backend_x86.yml the caller already built it and
+ # passes it as an input, so this job is skipped and the duplicate build avoided.
+ if: github.event_name == 'workflow_dispatch' || github.event_name == 'schedule'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
Review Comment:
The workflow relies on unpinned container image tags (e.g., `apache/gluten:centos-9-jdk17`). For stronger supply-chain security and reproducibility, pin these images by digest (or at least by an immutable version tag) so CI behavior can't change unexpectedly when the tag is updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3661214341
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -101,6 +106,70 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR
+ # touches high-signal Delta paths -- the Delta integration code
+ # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own
+ # files -- or carries the `run-delta-ci` opt-in label. Changes to general
+ # Velox/core/native code can also affect Delta offload but are touched
+ # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml
+ # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage
+ # down. NOTE: the label is read from the event that triggered this run, so add
+ # it before/with a push; labeling an already-finished run needs a new push.
+ delta-changes:
+ runs-on: ubuntu-22.04
+ outputs:
+ run_delta: ${{ steps.filter.outputs.run_delta }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ - name: Detect Delta-relevant changes / opt-in label
+ id: filter
+ env:
+ HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+ run: |
+ set -euo pipefail
+ # Opt-in label forces the suite even with no Delta-relevant path change.
+ if [ "$HAS_LABEL" = "true" ]; then
+ echo "run-delta-ci label present -> running Delta suite"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
+ fi
+ # Fail open if we can't determine the PR range (e.g. a non-PR trigger):
+ # never silently skip coverage.
+ if [ -z "${BASE_SHA:-}" ] || [ -z "${HEAD_SHA:-}" ]; then
+ echo "no PR base/head sha -> running Delta suite (fail-open)"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
+ fi
+ BASE=$(git merge-base "$BASE_SHA" "$HEAD_SHA" 2>/dev/null || echo "$BASE_SHA")
+ echo "diff base=$BASE head=$HEAD_SHA"
+ # High-signal Delta paths only: the Delta integration code
+ # (backends-velox/src-delta*), the Delta module, and this pipeline's own
+ # files. A change to general Velox/core/native code can also affect Delta
+ # offload, but those are touched constantly; per-PR we skip them (the
+ # nightly full run + the `run-delta-ci` label are the safety nets) to
+ # keep GHA usage down.
+ if git diff --name-only "$BASE" "$HEAD_SHA" | grep -Eq \
+ '^(\.github/workflows/velox_backend_x86\.yml|\.github/workflows/delta_spark_ut\.yml|\.github/workflows/util/delta-spark-ut/|gluten-delta/|backends-velox/src-delta)'; then
+ echo "Delta-relevant paths changed -> running Delta suite"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"
+ else
+ echo "No Delta-relevant paths changed and no opt-in label -> skipping Delta suite"
+ echo "run_delta=false" >> "$GITHUB_OUTPUT"
+ fi
Review Comment:
The `delta-changes` gate intends to “fail open” (run Delta suite) when the PR diff range can’t be computed, but with `set -euo pipefail` the `git diff --name-only ... | grep -Eq ...` pipeline can still fail (e.g. bad/missing commit objects or other git errors). In an `if ...; then` test, that failure will fall into the `else` branch and set `run_delta=false`, silently skipping coverage instead of failing open. Consider explicitly treating `git diff` failures as `run_delta=true`, then grep over the captured diff output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3617379617
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,208 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# init + shallow fetch resolves a tag, branch OR commit SHA in a single path
+# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the
+# destructive `rm -rf "$DELTA_DIR"` it required. `--` terminates options so a
+# DELTA_REF starting with `-` can't be misread as a git flag (this script is
+# workflow_dispatch-runnable with a user-supplied ref).
+git init -q "$DELTA_DIR"
+git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git
+git -C "$DELTA_DIR" fetch -q --depth 1 origin -- "$DELTA_REF"
+git -C "$DELTA_DIR" checkout -q FETCH_HEAD
+git -C "$DELTA_DIR" --no-pager log -1 --oneline
+echo "::endgroup::"
+
+echo "::group::Injecting Gluten bundle jar onto the spark project's TEST classpath"
+# The Gluten bundle jar must be on the spark project's TEST runtime classpath
+# (so DeltaSQLCommandTest can load org.apache.gluten.GlutenPlugin by name) but
+# NOT on the COMPILE classpath of `sparkV1`, which is the project that holds
+# Delta's main sources. The bundle's transitive contents include extra symbols
+# under `org.apache.spark.sql` that collide with Delta's main sources -- e.g.
+# MergeOutputGeneration.scala imports both `org.apache.spark.sql._` and
+# `org.apache.spark.sql.delta.ClassicColumnConversions._`, and would then fail
+# with `reference to expression is ambiguous`.
+#
+# sbt auto-scans `<baseDirectory>/lib` via `unmanagedBase`. Two relevant
+# projects in Delta v4.2.0 have a `lib/` baseDirectory:
+# - sparkV1: `project in file("spark")` -> spark/lib
+# - spark : `project in file("spark-unified")` -> spark-unified/lib
+# unmanagedJars are project-scoped (NOT inherited by dependents), so dropping
+# the bundle into spark-unified/lib/ adds it to the unified `spark` project's
+# Compile *and* Test classpaths -- but NOT to sparkV1's. That's exactly what
+# we want:
+# * sparkV1/Compile sees ONLY Delta's regular deps -> Delta main compiles.
+# * spark/Test/fullClasspath sees the bundle -> tests load GlutenPlugin.
+# (Verified empirically: with bundle only in spark-unified/lib/, sbt's
+# `show sparkV1/Compile/dependencyClasspath` excludes the bundle and
+# `show spark/Test/fullClasspath` includes it.)
+#
+# We deliberately do NOT also drop the bundle into spark/lib/, which is what
+# caused the previous compile failure: spark/lib/ is sparkV1's unmanagedBase,
+# and putting the bundle there would re-introduce the ambiguity errors.
+SPARK_UNIFIED_LIB="$DELTA_DIR/spark-unified/lib"
+mkdir -p "$SPARK_UNIFIED_LIB"
+cp "$GLUTEN_BUNDLE_JAR" "$SPARK_UNIFIED_LIB/gluten-velox-bundle.jar"
+ls -lh "$SPARK_UNIFIED_LIB"
+echo "::endgroup::"
+
+echo "::group::Patching DeltaSQLCommandTest to enable Gluten plugin"
+TARGET="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$TARGET" ]; then
+ echo "Expected file not found in Delta clone: $TARGET" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
+ exit 1
+fi
+cp "$PATCH_SOURCE" "$TARGET"
+echo "Patched $TARGET"
+echo "--- diff vs. upstream ---"
+git -C "$DELTA_DIR" --no-pager diff -- "spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala" || true
+echo "::endgroup::"
+
+# Delta's tests collect file-source scans by matching the concrete
+# `FileSourceScanExec` case class; Gluten offloads the scan to
+# DeltaScanTransformer, a `FileSourceScanLike` sibling, so those matches miss
+# (`scala.MatchError: List()`, empty partition filters, broken column-pruning /
+# scan-metric checks across many suites). delta-io/delta#7104 and #7105 widen the
+# matches to the shared `FileSourceScanLike` interface that both the vanilla and
+# Gluten scans implement (behavior-preserving for vanilla). Both are merged
+# upstream but land after the pinned DELTA_REF (v4.2.0), so apply them here; once
+# DELTA_REF includes a commit its cherry-pick is a clean no-op and the call can go.
+#
+# Depth-2 fetch brings each fix commit and its parent, which cherry-pick needs to
+# diff against (a depth-1 fetch grafts the parent away); `-n` stages the change
+# without requiring a committer identity.
+cherry_pick_delta_fix() {
+ local sha="$1" pr="$2"
+ echo "Cherry-picking delta-io/delta${pr}"
+ git -C "$DELTA_DIR" fetch --quiet --depth 2 origin "$sha"
+ git -C "$DELTA_DIR" cherry-pick -n "$sha"
+}
+
+echo "::group::Cherry-picking upstream Delta FileSourceScanLike test fixes"
+cherry_pick_delta_fix 46bd45d57eadd7e528002a0ae7bd36ce5a456eca "#7104 (ScanReportHelper.collectScans)"
+cherry_pick_delta_fix 959e00e15f41f56afc1c9bb95d160c55c6dc7068 "#7105 (9 more test suites)"
+echo "::endgroup::"
+
+echo "::group::Force-failing memory-hog DeletionVectorsSuite 2B-row tests"
+# Two DeletionVectorsSuite tests read from / delete from a 2-billion-row table.
+# Under the Gluten Velox bundle they balloon the forked test JVM to ~13G of
+# NATIVE memory (row-index materialization) and the kernel/cgroup OOM-kills it.
+# The dead fork then wedges sbt, hanging the whole shard until the workflow's
+# hang-watchdog dumps threads and kills it (~16 min wasted, and every suite
+# QUEUED AFTER it in that fork is skipped) -- see delta_spark_ut.yml.
+#
+# Rather than silently `ignore` these (easy to forget), we make them FAIL FAST
+# with a clear message: the gap stays visible in the test reports / baseline
+# until the native memory blow-up is fixed, at which point this patch should be
+# removed. NOTE: making the suite complete also un-skips the rest of the shard's
+# suite queue, so the known-failures baseline must be refreshed after this.
+#
+# ORDER MATTERS: keep this sed AFTER the cherry-picks above. #7105 also edits
+# DeletionVectorsSuite.scala, and git cherry-pick aborts (exit 128) when the work
+# tree has uncommitted edits to a file it touches.
+DVS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala"
+if [ ! -f "$DVS" ]; then
+ echo "Expected file not found in Delta clone: $DVS" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed." >&2
+ exit 1
+fi
+# Inject `fail(...)` as the first statement of each test body (the line ending
+# in `) {`). Delta sets no -Xfatal-warnings / dead-code warning, so the now-
+# unreachable original body compiles fine. Keep each injected line <100 chars:
+# Delta's scalastyle enforces a 100-char line length on test sources. The full
+# rationale lives in this comment, so the in-test message stays terse.
+sed -i 's#huge table: read from tables of 2B rows with existing DV of many zeros") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV read OOMs the test JVM; see setup-delta.sh")#' "$DVS"
+sed -i 's#number of rows from tables of 2B rows with DVs") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV delete OOMs the test JVM; see setup-delta.sh")#' "$DVS"
+INJECTED=$(grep -c "Gluten CI] Force-failed" "$DVS" || true)
Review Comment:
`sed -i` is GNU-sed specific; on macOS/BSD `sed` requires an explicit backup suffix (e.g. `-i ''`) and this script is documented as runnable locally. Using `-i.bak` (then removing the backup) keeps the script portable across GNU/BSD sed implementations.
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,208 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# init + shallow fetch resolves a tag, branch OR commit SHA in a single path
+# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the
+# destructive `rm -rf "$DELTA_DIR"` it required. `--` terminates options so a
+# DELTA_REF starting with `-` can't be misread as a git flag (this script is
+# workflow_dispatch-runnable with a user-supplied ref).
+git init -q "$DELTA_DIR"
+git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git
+git -C "$DELTA_DIR" fetch -q --depth 1 origin -- "$DELTA_REF"
+git -C "$DELTA_DIR" checkout -q FETCH_HEAD
+git -C "$DELTA_DIR" --no-pager log -1 --oneline
+echo "::endgroup::"
+
+echo "::group::Injecting Gluten bundle jar onto the spark project's TEST classpath"
+# The Gluten bundle jar must be on the spark project's TEST runtime classpath
+# (so DeltaSQLCommandTest can load org.apache.gluten.GlutenPlugin by name) but
+# NOT on the COMPILE classpath of `sparkV1`, which is the project that holds
+# Delta's main sources. The bundle's transitive contents include extra symbols
+# under `org.apache.spark.sql` that collide with Delta's main sources -- e.g.
+# MergeOutputGeneration.scala imports both `org.apache.spark.sql._` and
+# `org.apache.spark.sql.delta.ClassicColumnConversions._`, and would then fail
+# with `reference to expression is ambiguous`.
+#
+# sbt auto-scans `<baseDirectory>/lib` via `unmanagedBase`. Two relevant
+# projects in Delta v4.2.0 have a `lib/` baseDirectory:
+# - sparkV1: `project in file("spark")` -> spark/lib
+# - spark : `project in file("spark-unified")` -> spark-unified/lib
+# unmanagedJars are project-scoped (NOT inherited by dependents), so dropping
+# the bundle into spark-unified/lib/ adds it to the unified `spark` project's
+# Compile *and* Test classpaths -- but NOT to sparkV1's. That's exactly what
+# we want:
+# * sparkV1/Compile sees ONLY Delta's regular deps -> Delta main compiles.
+# * spark/Test/fullClasspath sees the bundle -> tests load GlutenPlugin.
+# (Verified empirically: with bundle only in spark-unified/lib/, sbt's
+# `show sparkV1/Compile/dependencyClasspath` excludes the bundle and
+# `show spark/Test/fullClasspath` includes it.)
+#
+# We deliberately do NOT also drop the bundle into spark/lib/, which is what
+# caused the previous compile failure: spark/lib/ is sparkV1's unmanagedBase,
+# and putting the bundle there would re-introduce the ambiguity errors.
+SPARK_UNIFIED_LIB="$DELTA_DIR/spark-unified/lib"
+mkdir -p "$SPARK_UNIFIED_LIB"
+cp "$GLUTEN_BUNDLE_JAR" "$SPARK_UNIFIED_LIB/gluten-velox-bundle.jar"
+ls -lh "$SPARK_UNIFIED_LIB"
+echo "::endgroup::"
+
+echo "::group::Patching DeltaSQLCommandTest to enable Gluten plugin"
+TARGET="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$TARGET" ]; then
+ echo "Expected file not found in Delta clone: $TARGET" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
+ exit 1
+fi
+cp "$PATCH_SOURCE" "$TARGET"
+echo "Patched $TARGET"
+echo "--- diff vs. upstream ---"
+git -C "$DELTA_DIR" --no-pager diff -- "spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala" || true
+echo "::endgroup::"
+
+# Delta's tests collect file-source scans by matching the concrete
+# `FileSourceScanExec` case class; Gluten offloads the scan to
+# DeltaScanTransformer, a `FileSourceScanLike` sibling, so those matches miss
+# (`scala.MatchError: List()`, empty partition filters, broken column-pruning /
+# scan-metric checks across many suites). delta-io/delta#7104 and #7105 widen the
+# matches to the shared `FileSourceScanLike` interface that both the vanilla and
+# Gluten scans implement (behavior-preserving for vanilla). Both are merged
+# upstream but land after the pinned DELTA_REF (v4.2.0), so apply them here; once
+# DELTA_REF includes a commit its cherry-pick is a clean no-op and the call can go.
+#
+# Depth-2 fetch brings each fix commit and its parent, which cherry-pick needs to
+# diff against (a depth-1 fetch grafts the parent away); `-n` stages the change
+# without requiring a committer identity.
+cherry_pick_delta_fix() {
+ local sha="$1" pr="$2"
+ echo "Cherry-picking delta-io/delta${pr}"
+ git -C "$DELTA_DIR" fetch --quiet --depth 2 origin "$sha"
+ git -C "$DELTA_DIR" cherry-pick -n "$sha"
+}
+
+echo "::group::Cherry-picking upstream Delta FileSourceScanLike test fixes"
+cherry_pick_delta_fix 46bd45d57eadd7e528002a0ae7bd36ce5a456eca "#7104 (ScanReportHelper.collectScans)"
+cherry_pick_delta_fix 959e00e15f41f56afc1c9bb95d160c55c6dc7068 "#7105 (9 more test suites)"
+echo "::endgroup::"
+
+echo "::group::Force-failing memory-hog DeletionVectorsSuite 2B-row tests"
+# Two DeletionVectorsSuite tests read from / delete from a 2-billion-row table.
+# Under the Gluten Velox bundle they balloon the forked test JVM to ~13G of
+# NATIVE memory (row-index materialization) and the kernel/cgroup OOM-kills it.
+# The dead fork then wedges sbt, hanging the whole shard until the workflow's
+# hang-watchdog dumps threads and kills it (~16 min wasted, and every suite
+# QUEUED AFTER it in that fork is skipped) -- see delta_spark_ut.yml.
+#
+# Rather than silently `ignore` these (easy to forget), we make them FAIL FAST
+# with a clear message: the gap stays visible in the test reports / baseline
+# until the native memory blow-up is fixed, at which point this patch should be
+# removed. NOTE: making the suite complete also un-skips the rest of the shard's
+# suite queue, so the known-failures baseline must be refreshed after this.
+#
+# ORDER MATTERS: keep this sed AFTER the cherry-picks above. #7105 also edits
+# DeletionVectorsSuite.scala, and git cherry-pick aborts (exit 128) when the work
+# tree has uncommitted edits to a file it touches.
+DVS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala"
+if [ ! -f "$DVS" ]; then
+ echo "Expected file not found in Delta clone: $DVS" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed." >&2
+ exit 1
+fi
+# Inject `fail(...)` as the first statement of each test body (the line ending
+# in `) {`). Delta sets no -Xfatal-warnings / dead-code warning, so the now-
+# unreachable original body compiles fine. Keep each injected line <100 chars:
+# Delta's scalastyle enforces a 100-char line length on test sources. The full
+# rationale lives in this comment, so the in-test message stays terse.
+sed -i 's#huge table: read from tables of 2B rows with existing DV of many zeros") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV read OOMs the test JVM; see setup-delta.sh")#' "$DVS"
+sed -i 's#number of rows from tables of 2B rows with DVs") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV delete OOMs the test JVM; see setup-delta.sh")#' "$DVS"
+INJECTED=$(grep -c "Gluten CI] Force-failed" "$DVS" || true)
+if [ "$INJECTED" -ne 2 ]; then
+ echo "ERROR: expected to force-fail 2 DeletionVectorsSuite tests but injected ${INJECTED}." >&2
+ echo "Their test names likely changed in Delta ref '${DELTA_REF}'; update setup-delta.sh." >&2
+ exit 1
+fi
+echo "Force-failed 2 DeletionVectorsSuite 2B-row tests (read + delete)."
+git -C "$DELTA_DIR" --no-pager diff -- "spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala" || true
+echo "::endgroup::"
+
+echo "::group::Disabling Delta scalastyle HeaderMatchesChecker"
+# Our reused DeltaSQLCommandTest carries Gluten's ASF-only license header, which
+# does not match Delta's HeaderMatchesChecker regex (the regex expects either a
+# Delta copyright block, or the ASF header followed by a Spark-modifications
+# block and the Delta copyright block). HeaderMatchesChecker is a file-level
+# checker that does NOT honor `// scalastyle:off` directives, so we instead
+# disable it globally in Delta's shared scalastyle-config.xml. The config is
+# applied via `ThisBuild / scalastyleConfig` in project/Checkstyle.scala, so a
+# single edit covers every sbt sub-project.
+SCALASTYLE_CONFIG="$DELTA_DIR/scalastyle-config.xml"
+if [ ! -f "$SCALASTYLE_CONFIG" ]; then
+ echo "Expected scalastyle config not found: $SCALASTYLE_CONFIG" >&2
+ exit 1
+fi
+sed -i \
+ 's|<check level="error" class="org.scalastyle.file.HeaderMatchesChecker" enabled="true">|<check level="error" class="org.scalastyle.file.HeaderMatchesChecker" enabled="false">|' \
+ "$SCALASTYLE_CONFIG"
+if ! grep -q '<check level="error" class="org.scalastyle.file.HeaderMatchesChecker" enabled="false">' "$SCALASTYLE_CONFIG"; then
Review Comment:
Same portability issue here: `sed -i` will fail on BSD/macOS. Switching to `sed -i.bak` and removing the backup file makes this edit cross-platform while keeping behavior identical on Linux.
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,446 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib artifact it already built, so the expensive
+ # native C++ build is NOT duplicated. That artifact lives in the CALLER's run (a
+ # called workflow runs as part of the caller run), so the jobs below download it
+ # by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. `spark_version` is the single source of truth for the
+ # Spark version: it drives the Gluten bundle profile (-Pspark-<v>), the bundle
+ # jar name, and Delta's -DsparkVersion, so the tests always run against a bundle
+ # built for the same Spark version (no separate value to keep in sync). Scala
+ # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from
+ # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a
+ # compatible delta_ref.
+ GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version }}
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version }}
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib and passes it as an input, so this
+ # job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
Review Comment:
This job clones and runs external code (delta-io/delta). `actions/checkout` persists the workflow token credentials into the workspace by default; disabling credential persistence reduces the risk of the token being read/used by code executed later in the job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3617361282
##########
.github/workflows/util/delta-spark-ut/run-delta-tests.sh:
##########
@@ -0,0 +1,240 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Runs the Delta `spark` module tests for one shard under the Gluten bundle:
+# arms a hang watchdog (thread-dumps + kills a wedged fork), invokes sbt
+# spark/test with the tuned JVM/heap flags, prints cgroup memory forensics, and
+# then gates the results against the baseline (compare-test-results.py). Extracted
+# from delta_spark_ut.yml so the workflow step stays readable.
+#
+# Driven by environment (set by the workflow step / job):
+# SHARD_ID - this shard's id (matrix.shard)
+# SPARK_VERSION - Delta -DsparkVersion value
+# UPDATE_BASELINE - 'true' -> gate seed mode; else enforce
+# FAIL_ON_FIXED - passed through to the gate
+# DELTA_SCALA_VERSION, NUM_SHARDS, TEST_PARALLELISM_COUNT, DELTA_TESTING
+# - test env (see the workflow step's `env:` block)
+# GITHUB_WORKSPACE - repo root (holds the Delta clone + util scripts)
+#
+# JAVA_TOOL_OPTIONS is set by sourcing java-test-args.sh (below), not the caller.
+
+set -euo pipefail
+export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+export PATH=$JAVA_HOME/bin:$PATH
+# Gluten/JDK17 test JVM flags (--add-opens + Netty property), shared with local
+# dev runs. Sets JAVA_TOOL_OPTIONS so it reaches the sbt launcher + forked JVMs.
+# shellcheck source=./java-test-args.sh
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/java-test-args.sh"
+cd "$GITHUB_WORKSPACE/delta"
+chmod +x build/sbt
+# Only run the unified `spark` sbt project, NOT `sparkGroup/test` --
+# `sparkGroup` aggregates many other projects (sparkV2, contribs,
+# sharing, connect*, ...) that are out of scope for this pipeline.
+#
+# JVM heap layout -- two memory consumers on the ~16G runner:
+# * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to
+# return idle memory during the (long) test phase via G1 periodic GC
+# (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the
+# busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent
+# forces each periodic GC to a full STW collection) that uncommits to a
+# tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low
+# -Xms512m floor.
+# Without this the idle launcher holds ~5.3G for the whole run; with
+# it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime
+# config, so they cannot affect the measured pass/fail signal.
+# * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command
+# below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends
+# so our -Xmx2G comes last and wins. Gluten offloads data to Velox
+# off-heap (capped at 2g via spark.memory.offHeap.size in the patched
+# DeltaSQLCommandTest), so the fork's heap need is modest. A larger
+# fork heap pushed the cgroup peak past the ~16G OOM threshold and the
+# kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G
+# keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is
+# analyzable.
+# `-u target/test-reports` enables ScalaTest's JUnit XML reporter so
+# every suite writes per-test results. Delta itself only configures
+# the console reporter (-oDF), so without this we'd have no machine-
+# readable results to gate on. The path is relative to the forked
+# test JVM's working dir (Test / baseDirectory = spark/), i.e.
+# delta/spark/target/test-reports/TEST-*.xml.
+#
+# We deliberately do NOT let an sbt non-zero exit (which fires on the
+# MANY expected Delta-on-Gluten failures) fail this step directly.
+# Instead the known-failures gate below decides pass/fail: the build
+# is green when the only failures are ones already recorded in the
+# baseline, and red on a genuine regression.
+set +e
+# --- hang watchdog ---------------------------------------------------
+# Shard 2 (and occasionally others) hangs indefinitely after a suite's
+# last test with no further output. ScalaTest's failAfter only wraps
+# individual test BODIES, so a wedge in suite teardown/afterAll -- or in
+# a non-interruptible native Velox/JNI call that ignores
+# Thread.interrupt() -- has no timeout and stalls until the 350-min job
+# limit with zero diagnostics. This watchdog dumps the forked test JVM's
+# threads (to the job log, and to a file for the artifact) once the test
+# output has been silent for too long, so the deadlock is diagnosable.
+SBT_LOG="/tmp/sbt-spark-test-shard-${SHARD_ID}.log"
+: > "$SBT_LOG"
+rm -f /tmp/sbt-done
+(
+ # CRITICAL: the step shell runs with `bash -eo pipefail`, which the
+ # subshell inherits. Without `set +e` here, ANY non-zero command --
+ # e.g. fork detection finding no match, or `kill`/`jps` returning
+ # non-zero -- silently kills this watchdog. That errexit kill (plus a
+ # /proc detection miss) once made the watchdog capture ZERO dumps. A
+ # diagnostic must never abort on a failed probe.
+ set +e +o pipefail
+ JSTACK="${JAVA_HOME}/bin/jstack"
+ JPS="${JAVA_HOME}/bin/jps"
+ silent_limit=900 # 15 min with no new test output => treat as hung
+ dumps=0
+ fork_pids() {
+ # The sbt test fork's main class is sbt.ForkMain. Prefer jps (reads
+ # the main class from hsperfdata, robust to sbt's @argfile launch);
+ # fall back to scanning /proc cmdline + @argfile.
+ "$JPS" -l 2>/dev/null | awk '/sbt\.ForkMain/ {print $1}'
+ local p cl arg
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] || continue
+ cl="$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)"
+ case "$cl" in *sbt.ForkMain*) echo "${p##*/}"; continue ;; esac
+ arg="$(printf '%s' "$cl" | tr ' ' '\n' | sed -n 's/^@//p' | head -1)"
+ [ -n "$arg" ] && [ -f "$arg" ] && grep -qa 'sbt\.ForkMain' "$arg" 2>/dev/null \
+ && echo "${p##*/}"
+ done
+ }
+ all_java_pids() {
+ "$JPS" -q 2>/dev/null
+ local p
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] && echo "${p##*/}"
+ done
+ }
+ echo "HANG WATCHDOG armed: dumps the test JVM after ${silent_limit}s of output silence"
+ hb=0
+ while [ ! -f /tmp/sbt-done ]; do
+ sleep 60
+ [ -f "$SBT_LOG" ] || continue
+ now=$(date +%s)
+ mtime=$(stat -c %Y "$SBT_LOG" 2>/dev/null || echo "$now")
+ silent=$(( now - mtime ))
+ # Per-minute memory profile: heap tuning proved the ~16G OOM peak is
+ # NATIVE-driven, so log which JVM (sbt launcher vs fork) actually grows
+ # toward it -- the last lines before a hang reveal the real hog to cut.
+ # Read /proc directly (no `ps` dependency in the minimal container).
+ memnow=$(awk '{printf "%.2fG",$1/1073741824}' /sys/fs/cgroup/memory.current 2>/dev/null)
+ jvmrss=""
+ for mp in $(all_java_pids 2>/dev/null | sort -un); do
+ r=$(awk '/^VmRSS:/{print $2}' "/proc/$mp/status" 2>/dev/null)
+ [ -n "$r" ] && jvmrss="$jvmrss $(( r / 1024 ))M(p$mp)"
+ done
+ echo "MEM cgroup=${memnow} JVMs=[${jvmrss# }]"
+ hb=$(( hb + 1 ))
+ # Heartbeat every ~5 min so we can SEE the watchdog is alive (and how
+ # long the test has been silent) without waiting for a hang.
+ [ $(( hb % 5 )) -eq 0 ] && echo "HANG WATCHDOG: alive; last test output ${silent}s ago"
+ if [ "$silent" -ge "$silent_limit" ] && [ "$dumps" -lt 3 ]; then
+ dumps=$(( dumps + 1 ))
+ pids="$(fork_pids | sort -un)"
+ # Safety net: if the fork JVM cannot be pinpointed, dump EVERY JVM.
+ [ -n "$pids" ] || pids="$(all_java_pids | sort -un)"
+ echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $pids))"
+ [ -n "$pids" ] || echo "HANG WATCHDOG: no java process found to dump"
+ for pid in $pids; do
+ # SIGQUIT makes the JVM print a full thread dump to its OWN stderr,
+ # which sbt relays into the test log via the SAME stream as test
+ # output -- so it lands in the job log even when a separately
+ # spawned jstack child's output would be buffered/lost. Also write
+ # jstack to a file for the per-shard artifact.
+ echo "----- SIGQUIT + jstack pid ${pid} -----"
+ kill -QUIT "$pid" 2>/dev/null || echo "HANG WATCHDOG: kill -QUIT failed for pid ${pid}"
+ timeout 120 "$JSTACK" -l "$pid" > "/tmp/threaddump-shard-${SHARD_ID}-${dumps}-${pid}.txt" 2>&1 \
+ || echo "HANG WATCHDOG: jstack failed/timed out for pid ${pid}"
+ done
+ echo "::endgroup::"
+ # The dump is now captured (job log via SIGQUIT + artifact via
+ # jstack file). A hung JVM otherwise stalls the whole shard until
+ # the 350-min job timeout AND keeps the job log frozen so the dump
+ # never becomes reachable. So KILL the wedged JVM(s): the suite
+ # fails fast (acceptable -- errors are expected; only an
+ # unrecoverable hang blocks CI), the job proceeds/ends, and the log
+ # + artifacts flush. Give SIGQUIT a moment to print first.
+ sleep 20
+ echo "HANG WATCHDOG: killing wedged JVM(s) to unblock the shard: $(printf '%s ' $pids)"
+ for pid in $pids; do kill -KILL "$pid" 2>/dev/null; done
Review Comment:
Good catch — you're reading it right, and this was a real gap. A hang+kill left the killed fork's running suite plus everything queued behind it in that JVM unrun and unreported, and since we ignore sbt's exit code the gate only judged the suites that reported, so the shard could go green. Fixed as you suggested: the watchdog now `touch`es a marker file when it kills a wedged fork, and after sbt returns the script fails the shard (`::error::` + `exit 1`) if the marker exists. A watchdog kill is an abnormal run, so it gets a red X. (c21e3ec22)
##########
.github/workflows/util/delta-spark-ut/run-delta-tests.sh:
##########
@@ -0,0 +1,240 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Runs the Delta `spark` module tests for one shard under the Gluten bundle:
+# arms a hang watchdog (thread-dumps + kills a wedged fork), invokes sbt
+# spark/test with the tuned JVM/heap flags, prints cgroup memory forensics, and
+# then gates the results against the baseline (compare-test-results.py). Extracted
+# from delta_spark_ut.yml so the workflow step stays readable.
+#
+# Driven by environment (set by the workflow step / job):
+# SHARD_ID - this shard's id (matrix.shard)
+# SPARK_VERSION - Delta -DsparkVersion value
+# UPDATE_BASELINE - 'true' -> gate seed mode; else enforce
+# FAIL_ON_FIXED - passed through to the gate
+# DELTA_SCALA_VERSION, NUM_SHARDS, TEST_PARALLELISM_COUNT, DELTA_TESTING
+# - test env (see the workflow step's `env:` block)
+# GITHUB_WORKSPACE - repo root (holds the Delta clone + util scripts)
+#
+# JAVA_TOOL_OPTIONS is set by sourcing java-test-args.sh (below), not the caller.
+
+set -euo pipefail
+export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+export PATH=$JAVA_HOME/bin:$PATH
+# Gluten/JDK17 test JVM flags (--add-opens + Netty property), shared with local
+# dev runs. Sets JAVA_TOOL_OPTIONS so it reaches the sbt launcher + forked JVMs.
+# shellcheck source=./java-test-args.sh
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/java-test-args.sh"
+cd "$GITHUB_WORKSPACE/delta"
+chmod +x build/sbt
+# Only run the unified `spark` sbt project, NOT `sparkGroup/test` --
+# `sparkGroup` aggregates many other projects (sparkV2, contribs,
+# sharing, connect*, ...) that are out of scope for this pipeline.
+#
+# JVM heap layout -- two memory consumers on the ~16G runner:
+# * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to
+# return idle memory during the (long) test phase via G1 periodic GC
+# (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the
+# busy fork doesn't suppress it; -XX:-G1PeriodicGCInvokesConcurrent
+# forces each periodic GC to a full STW collection) that uncommits to a
+# tight free ratio (Min/MaxHeapFreeRatio 5/15, JEP 346) above a low
+# -Xms512m floor.
+# Without this the idle launcher holds ~5.3G for the whole run; with
+# it, it drops back to ~1-2G. These flags touch no Gluten/Spark runtime
+# config, so they cannot affect the measured pass/fail signal.
+# * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command
+# below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends
+# so our -Xmx2G comes last and wins. Gluten offloads data to Velox
+# off-heap (capped at 2g via spark.memory.offHeap.size in the patched
+# DeltaSQLCommandTest), so the fork's heap need is modest. A larger
+# fork heap pushed the cgroup peak past the ~16G OOM threshold and the
+# kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G
+# keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is
+# analyzable.
+# `-u target/test-reports` enables ScalaTest's JUnit XML reporter so
+# every suite writes per-test results. Delta itself only configures
+# the console reporter (-oDF), so without this we'd have no machine-
+# readable results to gate on. The path is relative to the forked
+# test JVM's working dir (Test / baseDirectory = spark/), i.e.
+# delta/spark/target/test-reports/TEST-*.xml.
+#
+# We deliberately do NOT let an sbt non-zero exit (which fires on the
+# MANY expected Delta-on-Gluten failures) fail this step directly.
+# Instead the known-failures gate below decides pass/fail: the build
+# is green when the only failures are ones already recorded in the
+# baseline, and red on a genuine regression.
+set +e
+# --- hang watchdog ---------------------------------------------------
+# Shard 2 (and occasionally others) hangs indefinitely after a suite's
+# last test with no further output. ScalaTest's failAfter only wraps
+# individual test BODIES, so a wedge in suite teardown/afterAll -- or in
+# a non-interruptible native Velox/JNI call that ignores
+# Thread.interrupt() -- has no timeout and stalls until the 350-min job
+# limit with zero diagnostics. This watchdog dumps the forked test JVM's
+# threads (to the job log, and to a file for the artifact) once the test
+# output has been silent for too long, so the deadlock is diagnosable.
+SBT_LOG="/tmp/sbt-spark-test-shard-${SHARD_ID}.log"
+: > "$SBT_LOG"
+rm -f /tmp/sbt-done
+(
+ # CRITICAL: the step shell runs with `bash -eo pipefail`, which the
+ # subshell inherits. Without `set +e` here, ANY non-zero command --
+ # e.g. fork detection finding no match, or `kill`/`jps` returning
+ # non-zero -- silently kills this watchdog. That errexit kill (plus a
+ # /proc detection miss) once made the watchdog capture ZERO dumps. A
+ # diagnostic must never abort on a failed probe.
+ set +e +o pipefail
+ JSTACK="${JAVA_HOME}/bin/jstack"
+ JPS="${JAVA_HOME}/bin/jps"
+ silent_limit=900 # 15 min with no new test output => treat as hung
+ dumps=0
+ fork_pids() {
+ # The sbt test fork's main class is sbt.ForkMain. Prefer jps (reads
+ # the main class from hsperfdata, robust to sbt's @argfile launch);
+ # fall back to scanning /proc cmdline + @argfile.
+ "$JPS" -l 2>/dev/null | awk '/sbt\.ForkMain/ {print $1}'
+ local p cl arg
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] || continue
+ cl="$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)"
+ case "$cl" in *sbt.ForkMain*) echo "${p##*/}"; continue ;; esac
+ arg="$(printf '%s' "$cl" | tr ' ' '\n' | sed -n 's/^@//p' | head -1)"
+ [ -n "$arg" ] && [ -f "$arg" ] && grep -qa 'sbt\.ForkMain' "$arg" 2>/dev/null \
+ && echo "${p##*/}"
+ done
+ }
+ all_java_pids() {
+ "$JPS" -q 2>/dev/null
+ local p
+ for p in /proc/[0-9]*; do
+ [ "$(cat "$p/comm" 2>/dev/null)" = "java" ] && echo "${p##*/}"
+ done
+ }
+ echo "HANG WATCHDOG armed: dumps the test JVM after ${silent_limit}s of output silence"
+ hb=0
+ while [ ! -f /tmp/sbt-done ]; do
+ sleep 60
+ [ -f "$SBT_LOG" ] || continue
+ now=$(date +%s)
+ mtime=$(stat -c %Y "$SBT_LOG" 2>/dev/null || echo "$now")
+ silent=$(( now - mtime ))
+ # Per-minute memory profile: heap tuning proved the ~16G OOM peak is
+ # NATIVE-driven, so log which JVM (sbt launcher vs fork) actually grows
+ # toward it -- the last lines before a hang reveal the real hog to cut.
+ # Read /proc directly (no `ps` dependency in the minimal container).
+ memnow=$(awk '{printf "%.2fG",$1/1073741824}' /sys/fs/cgroup/memory.current 2>/dev/null)
+ jvmrss=""
+ for mp in $(all_java_pids 2>/dev/null | sort -un); do
+ r=$(awk '/^VmRSS:/{print $2}' "/proc/$mp/status" 2>/dev/null)
+ [ -n "$r" ] && jvmrss="$jvmrss $(( r / 1024 ))M(p$mp)"
+ done
+ echo "MEM cgroup=${memnow} JVMs=[${jvmrss# }]"
+ hb=$(( hb + 1 ))
+ # Heartbeat every ~5 min so we can SEE the watchdog is alive (and how
+ # long the test has been silent) without waiting for a hang.
+ [ $(( hb % 5 )) -eq 0 ] && echo "HANG WATCHDOG: alive; last test output ${silent}s ago"
+ if [ "$silent" -ge "$silent_limit" ] && [ "$dumps" -lt 3 ]; then
+ dumps=$(( dumps + 1 ))
+ pids="$(fork_pids | sort -un)"
+ # Safety net: if the fork JVM cannot be pinpointed, dump EVERY JVM.
+ [ -n "$pids" ] || pids="$(all_java_pids | sort -un)"
+ echo "::group::HANG WATCHDOG: test output silent ${silent}s -- thread dump #${dumps} (pids:$(printf ' %s' $pids))"
+ [ -n "$pids" ] || echo "HANG WATCHDOG: no java process found to dump"
Review Comment:
Agreed. I split the dump set from the kill set: we still dump *every* JVM for diagnostics when no `sbt.ForkMain` fork can be pinpointed, but we now only KILL matched fork pids. During a pre-fork stall (dependency resolution or a cold-cache compile) the watchdog dumps and leaves sbt running instead of killing the launcher, so we don't waste the slot on a confusing "compile/launch failure". Only a real wedged fork is killed (and only then is the fail-the-shard marker set). I also made the per-episode dump/kill budget reset when output resumes, so a transient pre-fork stall we dump-but-don't-kill can't starve the budget for a real fork hang later. (c21e3ec22)
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,747 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline file exists but is empty (not yet bootstrapped) the mode
+ automatically degrades to ``seed`` so the first run is never spuriously red.
+ A *missing* ``--known-failures`` file is treated as a configuration error
+ (the gate fails) so a mis-referenced path can't silently pass.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard). Pass ``--expected-shards N``
+ to fail when fewer than ``N`` shards contributed gate lists (a shard that
+ died before writing them), so an incomplete baseline is never produced.
+
+Flaky quarantine (``--flaky-tests``)
+ Some Delta-on-Gluten failures are non-deterministic (e.g. a native bug that
+ only triggers on certain runtime plans), so they are neither a stable pass
+ nor a stable failure and cannot live in the baseline: baselining them turns
+ the gate red on every run where they pass, and leaving them out turns it red
+ on every run where they fail. ``flaky-tests.txt`` quarantines them -- a
+ quarantined test never counts as a regression (when it fails) nor as
+ now-passing (when it passes), and is excluded from the regenerated baseline.
+ Its SUITE is an fnmatch glob (so one line covers a root-cause family across
+ generated suite variants); its TEST name is matched exactly.
+
+Flaky quarantine by error signature (``--flaky-error-patterns``)
+ When a failure is caused by a known nondeterministic bug that surfaces on a
+ *different test each run* (e.g. the native Delta DV bitmap row-index error),
+ matching by test name is whack-a-mole. ``flaky-error-patterns.txt`` instead
+ quarantines by root cause: each line is a regex matched against a failed
+ test's <failure>/<error> text, and any failure that matches is treated as
+ flaky regardless of which test it landed on.
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import fnmatch
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+
+class NoReportsError(RuntimeError):
+ """Raised when no JUnit <testsuite> elements are found under reports_dir."""
+
+
+class CorruptReportError(NoReportsError):
+ """Raised when an expected JUnit report file (TEST-*.xml) fails to parse.
+
+ Subclasses NoReportsError so the enforce/seed handler treats a truncated
+ report as a hard data error (exit 2) instead of silently dropping the
+ suite's results and letting the gate pass on partial data.
+ """
+
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def make_is_flaky(flaky_entries):
+ """Build a predicate that matches a (suite, test) tuple against flaky entries.
+
+ A flaky entry quarantines a test whose failure is known to be non-deterministic
+ (see flaky-tests.txt). The entry's SUITE is treated as an fnmatch glob so a
+ single line can cover a root-cause family across generated suite variants
+ (e.g. ``*DVs*Suite`` matches every deletion-vector merge suite, ``*`` matches
+ any suite); the TEST name is matched exactly (test names are freeform and may
+ contain glob metacharacters, so they are never globbed).
+ """
+ exact = set()
+ globbed = []
+ for suite, test in flaky_entries:
+ if any(ch in suite for ch in "*?["):
+ globbed.append((suite, test))
+ else:
+ exact.add((suite, test))
+
+ def is_flaky(entry):
+ if entry in exact:
+ return True
+ suite, test = entry
+ for glob_suite, glob_test in globbed:
+ if test == glob_test and fnmatch.fnmatchcase(suite, glob_suite):
+ return True
+ return False
+
+ return is_flaky
+
+
+def load_patterns(path):
+ """Load flaky-error regex patterns from a file (one per line).
+
+ Blank lines and ``#`` comments are ignored. Each remaining line is compiled
+ as a case-sensitive regex. These match the FAILURE TEXT of a failed test
+ (its JUnit <failure>/<error> message + stack), so a test that fails with a
+ known-nondeterministic native error (e.g. the Delta DV bitmap row-index bug)
+ can be quarantined by root cause instead of by exact test name.
+ """
+ patterns = []
+ if not path or not os.path.exists(path):
+ return patterns
+ with open(path, encoding="utf-8") as fh:
+ for line in fh:
+ line = line.rstrip("\n")
+ if not line.strip() or line.lstrip().startswith("#"):
+ continue
+ patterns.append(re.compile(line))
+ return patterns
+
+
+def make_signature_matcher(patterns):
+ """Return a predicate matching a failure text against any flaky-error pattern."""
+
+ def matches(text):
+ if not text:
+ return False
+ return any(p.search(text) for p in patterns)
+
+ return matches
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def _failure_text(tc):
+ """Concatenate the message attribute + body text of a testcase's
+ <failure>/<error> children, for error-signature matching."""
+ parts = []
+ for c in tc:
+ if c.tag.split("}")[-1] in ("failure", "error"):
+ msg = c.get("message")
+ if msg:
+ parts.append(msg)
+ if c.text:
+ parts.append(c.text)
+ return "\n".join(parts)
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped, fail_texts). The first three are sets of
+ (suite, test) tuples; fail_texts maps each failed (suite, test) to its
+ combined <failure>/<error> message + stack text (used for error-signature
+ quarantine). A test is 'failed' if its <testcase> has a <failure> or <error>
+ child, 'skipped' if it has a <skipped> child, otherwise 'passed'. Suite-level
+ aborts (a <testsuite> reporting errors/failures with no failing <testcase>)
+ are recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+ fail_texts = {}
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ # A TEST-*.xml that fails to parse is almost always a report truncated
+ # when a forked test JVM was killed mid-write (e.g. OOM). Silently
+ # skipping it drops that suite's results and could let the gate go
+ # green on partial data, so fail hard for report files. Other XML that
+ # merely matched the broad `target/**` glob is still skipped.
+ if os.path.basename(xml_file).startswith("TEST-"):
+ raise CorruptReportError(
+ "corrupt or truncated JUnit report {}: {}. Refusing to "
+ "evaluate the gate on partial data.".format(xml_file, exc)
+ )
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = (suite, name)
Review Comment:
Nice one — subtle but real. Added a `normalize_key(suite, test)` helper that runs the XML-derived key through the same round-trip the baseline uses (`write_entries` collapses CR/LF in the test name to spaces; `parse_entry` strips the whole `suite#test` line), applied at both key sites (the testcase key and the suite-abort key). It's a no-op for normal names, but now a name with a trailing newline/whitespace normalizes to match a line pasted from the gate's `REGRESSION` output, so it's actually suppressible. Verified with an enforce-mode run using a report whose testcase `name` carries a trailing newline. (c21e3ec22)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5120845176
@felipepessoto thanks for the patch, could we also remove the Delta tests from Spark-40/41 with this refactor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3554139127
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,747 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline file exists but is empty (not yet bootstrapped) the mode
+ automatically degrades to ``seed`` so the first run is never spuriously red.
+ A *missing* ``--known-failures`` file is treated as a configuration error
+ (the gate fails) so a mis-referenced path can't silently pass.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard). Pass ``--expected-shards N``
+ to fail when fewer than ``N`` shards contributed gate lists (a shard that
+ died before writing them), so an incomplete baseline is never produced.
+
+Flaky quarantine (``--flaky-tests``)
+ Some Delta-on-Gluten failures are non-deterministic (e.g. a native bug that
+ only triggers on certain runtime plans), so they are neither a stable pass
+ nor a stable failure and cannot live in the baseline: baselining them turns
+ the gate red on every run where they pass, and leaving them out turns it red
+ on every run where they fail. ``flaky-tests.txt`` quarantines them -- a
+ quarantined test never counts as a regression (when it fails) nor as
+ now-passing (when it passes), and is excluded from the regenerated baseline.
+ Its SUITE is an fnmatch glob (so one line covers a root-cause family across
+ generated suite variants); its TEST name is matched exactly.
+
+Flaky quarantine by error signature (``--flaky-error-patterns``)
+ When a failure is caused by a known nondeterministic bug that surfaces on a
+ *different test each run* (e.g. the native Delta DV bitmap row-index error),
+ matching by test name is whack-a-mole. ``flaky-error-patterns.txt`` instead
+ quarantines by root cause: each line is a regex matched against a failed
+ test's <failure>/<error> text, and any failure that matches is treated as
+ flaky regardless of which test it landed on.
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import fnmatch
+import glob
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+
+class NoReportsError(RuntimeError):
+ """Raised when no JUnit <testsuite> elements are found under reports_dir."""
+
+
+class CorruptReportError(NoReportsError):
+ """Raised when an expected JUnit report file (TEST-*.xml) fails to parse.
+
+ Subclasses NoReportsError so the enforce/seed handler treats a truncated
+ report as a hard data error (exit 2) instead of silently dropping the
+ suite's results and letting the gate pass on partial data.
+ """
+
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def make_is_flaky(flaky_entries):
+ """Build a predicate that matches a (suite, test) tuple against flaky entries.
+
+ A flaky entry quarantines a test whose failure is known to be non-deterministic
+ (see flaky-tests.txt). The entry's SUITE is treated as an fnmatch glob so a
+ single line can cover a root-cause family across generated suite variants
+ (e.g. ``*DVs*Suite`` matches every deletion-vector merge suite, ``*`` matches
+ any suite); the TEST name is matched exactly (test names are freeform and may
+ contain glob metacharacters, so they are never globbed).
+ """
+ exact = set()
+ globbed = []
+ for suite, test in flaky_entries:
+ if any(ch in suite for ch in "*?["):
+ globbed.append((suite, test))
+ else:
+ exact.add((suite, test))
+
+ def is_flaky(entry):
+ if entry in exact:
+ return True
+ suite, test = entry
+ for glob_suite, glob_test in globbed:
+ if test == glob_test and fnmatch.fnmatchcase(suite, glob_suite):
+ return True
+ return False
+
+ return is_flaky
+
+
+def load_patterns(path):
+ """Load flaky-error regex patterns from a file (one per line).
+
+ Blank lines and ``#`` comments are ignored. Each remaining line is compiled
+ as a case-sensitive regex. These match the FAILURE TEXT of a failed test
+ (its JUnit <failure>/<error> message + stack), so a test that fails with a
+ known-nondeterministic native error (e.g. the Delta DV bitmap row-index bug)
+ can be quarantined by root cause instead of by exact test name.
+ """
+ patterns = []
+ if not path or not os.path.exists(path):
+ return patterns
+ with open(path, encoding="utf-8") as fh:
+ for line in fh:
+ line = line.rstrip("\n")
+ if not line.strip() or line.lstrip().startswith("#"):
+ continue
+ patterns.append(re.compile(line))
+ return patterns
+
+
+def make_signature_matcher(patterns):
+ """Return a predicate matching a failure text against any flaky-error pattern."""
+
+ def matches(text):
+ if not text:
+ return False
+ return any(p.search(text) for p in patterns)
+
+ return matches
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def _failure_text(tc):
+ """Concatenate the message attribute + body text of a testcase's
+ <failure>/<error> children, for error-signature matching."""
+ parts = []
+ for c in tc:
+ if c.tag.split("}")[-1] in ("failure", "error"):
+ msg = c.get("message")
+ if msg:
+ parts.append(msg)
+ if c.text:
+ parts.append(c.text)
+ return "\n".join(parts)
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped, fail_texts). The first three are sets of
+ (suite, test) tuples; fail_texts maps each failed (suite, test) to its
+ combined <failure>/<error> message + stack text (used for error-signature
+ quarantine). A test is 'failed' if its <testcase> has a <failure> or <error>
+ child, 'skipped' if it has a <skipped> child, otherwise 'passed'. Suite-level
+ aborts (a <testsuite> reporting errors/failures with no failing <testcase>)
+ are recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+ fail_texts = {}
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ # A TEST-*.xml that fails to parse is almost always a report truncated
+ # when a forked test JVM was killed mid-write (e.g. OOM). Silently
+ # skipping it drops that suite's results and could let the gate go
+ # green on partial data, so fail hard for report files. Other XML that
+ # merely matched the broad `target/**` glob is still skipped.
+ if os.path.basename(xml_file).startswith("TEST-"):
+ raise CorruptReportError(
+ "corrupt or truncated JUnit report {}: {}. Refusing to "
+ "evaluate the gate on partial data.".format(xml_file, exc)
+ )
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = (suite, name)
+ tags = _child_local_tags(tc)
+ if "failure" in tags or "error" in tags:
+ failed.add(key)
+ suite_has_failing_tc = True
+ fail_texts[key] = _failure_text(tc)
+ elif "skipped" in tags:
+ skipped.add(key)
+ else:
+ passed.add(key)
+
+ # Suite-level abort: counters say something failed but no testcase
+ # carried the failure (the suite blew up in beforeAll/constructor).
+ # Record a
+ # synthetic entry so the regression is visible.
+ try:
+ errors = int(ts.get("errors", "0") or "0")
+ failures = int(ts.get("failures", "0") or "0")
+ except ValueError:
+ errors = failures = 0
+ if (errors + failures) > 0 and not suite_has_failing_tc:
+ failed.add((suite_name, SUITE_ABORTED))
+
+ if not parsed_any:
+ raise NoReportsError(
+ "No JUnit <testsuite> elements found under {}. The test reports are "
+ "missing or in an unexpected format -- refusing to evaluate the gate "
+ "on an empty result set (this would otherwise pass silently).".format(
+ reports_dir
+ )
+ )
+
+ # A test can't be both passed and failed; failure wins. Skipped only counts
+ # if the test was not otherwise seen (e.g. retried).
+ passed -= failed
+ skipped -= failed
+ skipped -= passed
+ return passed, failed, skipped, fail_texts
+
+
+# --------------------------------------------------------------------------- #
+# Reporting helpers
+# --------------------------------------------------------------------------- #
+def _summary_sink():
+ """Return a writer that mirrors to GITHUB_STEP_SUMMARY when available."""
+ path = os.environ.get("GITHUB_STEP_SUMMARY")
+ handle = open(path, "a", encoding="utf-8") if path else None
+
+ def write(line=""):
+ print(line)
+ if handle:
+ handle.write(line + "\n")
+
+ return write, handle
+
+
+def _print_block(write, title, entries, limit=50):
+ write("")
+ write("### {} ({})".format(title, len(entries)))
+ if not entries:
+ return
+ write("")
+ write("```")
+ for i, (suite, test) in enumerate(sorted(entries)):
+ if i >= limit:
+ write("... and {} more".format(len(entries) - limit))
+ break
+ write(format_entry(suite, test))
+ write("```")
+
+
+# --------------------------------------------------------------------------- #
+# Modes
+# --------------------------------------------------------------------------- #
+def run_enforce(args):
+ # In enforce mode a missing baseline file would make load_entries() return an
+ # empty set, silently degrading to seed mode and passing the gate without
+ # enforcing anything. Treat a missing path as a configuration error; an
+ # existing-but-empty file is still allowed (it legitimately seeds).
+ if args.mode == "enforce" and (
+ not args.known_failures or not os.path.exists(args.known_failures)
+ ):
+ eprint(
+ "ERROR: --known-failures '{}' does not exist. In enforce mode the "
+ "baseline file must exist (an existing-but-empty file is allowed and "
+ "triggers seed mode). Refusing to silently pass.".format(
+ args.known_failures
+ )
+ )
+ return 2
+ baseline = load_entries(args.known_failures)
+ name_flaky = make_is_flaky(load_entries(args.flaky_tests))
+ sig_matches = make_signature_matcher(load_patterns(args.flaky_error_patterns))
+ try:
+ passed, failed, skipped, fail_texts = parse_reports(args.reports_dir)
+ except NoReportsError as exc:
+ eprint("ERROR: {}".format(exc))
+ return 2
+
+ # A test is quarantined if its NAME is in flaky-tests.txt, or its failure
+ # TEXT matches a flaky-error signature (e.g. the native DV bitmap row-index
+ # bug that hits a different DV-merge test each run).
+ def sig_flaky(e):
+ return e in fail_texts and sig_matches(fail_texts[e])
+
+ def flaky_is(e):
+ return name_flaky(e) or sig_flaky(e)
+
+ # Always emit this shard's artifacts for the aggregation job. Signature-flaky
+ # failures are dropped from failures-out: the aggregate job works off these
+ # text-less lists and cannot re-derive the signature match, so excluding them
+ # here keeps the regenerated baseline from absorbing a flaky failure. Name-
+ # flaky entries stay (the aggregate re-filters them via flaky-tests.txt).
+ if args.failures_out:
+ write_entries(args.failures_out, {e for e in failed if not sig_flaky(e)})
+ if args.ran_out:
+ write_entries(args.ran_out, passed | failed)
+
+ write, handle = _summary_sink()
+ try:
+ seeding = args.mode == "seed" or not baseline
+ if seeding and args.mode != "seed":
+ write(
+ "> NOTE: baseline `{}` is empty -- running in SEED mode "
+ "(no failures will be enforced). Bootstrap the baseline from "
+ "the aggregated artifact, commit it, then enforcement begins.".format(
+ args.known_failures
+ )
+ )
+
+ write(
+ "## Delta-on-Gluten test gate -- shard {}".format(
+ os.environ.get("SHARD_ID", "?")
+ )
+ )
+ write("")
+ write("| Category | Count |")
+ write("|---|---:|")
+ write("| Ran (pass+fail) | {} |".format(len(passed) + len(failed)))
+ write("| Passed | {} |".format(len(passed)))
+ write("| Failed | {} |".format(len(failed)))
+ write("| Skipped | {} |".format(len(skipped)))
+ write("| Baseline (known failures) | {} |".format(len(baseline)))
+
+ if seeding:
+ write("")
+ write(
+ "Seed mode: recorded {} failing test(s) for this shard. "
+ "Nothing enforced.".format(len(failed))
+ )
+ return 0
+
+ regressions = {e for e in (failed - baseline) if not flaky_is(e)}
+ quarantined = {e for e in (failed - baseline) if flaky_is(e)}
+ # `- flaky` on fixed is defensive: flaky tests are excluded from the
+ # regenerated baseline (aggregate mode), so a flaky test should never be in
+ # `baseline` in the first place -- but if one slips in, don't let its
+ # non-deterministic pass trip the now-passing gate.
+ fixed = {e for e in (baseline & passed) if not flaky_is(e)}
+ expected = failed & baseline
+
+ write("")
+ write("| Gate result | Count |")
+ write("|---|---:|")
+ write("| Expected failures (in baseline) | {} |".format(len(expected)))
+ write("| **Regressions (new failures)** | {} |".format(len(regressions)))
+ write("| Now-passing (remove from baseline) | {} |".format(len(fixed)))
+ write("| Quarantined flaky failures (ignored) | {} |".format(len(quarantined)))
+
+ _print_block(
+ write, "Regressions -- new failures NOT in the baseline", regressions
+ )
+ if regressions:
+ write("")
+ write(
+ "These tests were not previously known to fail. Either fix "
+ "the regression, or (if it is a genuinely new expected "
+ "failure) add the lines above to `known-failures.txt`."
+ )
+
+ _print_block(
+ write,
+ "Quarantined flaky failures -- ignored (flaky-tests.txt + flaky-error-patterns.txt)",
+ quarantined,
+ )
+
+ if args.fail_on_fixed:
+ _print_block(
+ write, "Now-passing -- delete these lines from the baseline", fixed
+ )
+
+ exit_code = 0
+ if regressions:
+ for suite, test in sorted(regressions):
+ eprint("::error::REGRESSION {}".format(format_entry(suite, test)))
+ exit_code = 1
+ if args.fail_on_fixed and fixed:
+ for suite, test in sorted(fixed):
+ eprint(
+ "::error::NOW-PASSING (remove from baseline) {}".format(
+ format_entry(suite, test)
+ )
+ )
+ exit_code = 1
+
+ if exit_code == 0:
+ write("")
+ write("All failures are expected (in the baseline). Gate passed.")
+ return exit_code
+ finally:
+ if handle:
+ handle.close()
+
+
+def _shard_ids(files, prefix):
+ """Return the set of shard ids from gate-list filenames.
+
+ Gate lists are named ``<prefix><shard-id>.txt`` (e.g. ``failures-shard-0.txt``,
+ ``ran-shard-0.txt``); this extracts the ``<shard-id>`` token so aggregate mode
+ can count how many shards contributed. Matching is on the basename, so nested
+ download dirs are fine.
+ """
+ ids = set()
+ pat = re.compile(r"^" + re.escape(prefix) + r"(.+)\.txt$")
+ for f in files:
+ m = pat.match(os.path.basename(f))
+ if m:
+ ids.add(m.group(1))
+ return ids
+
+
+def run_aggregate(args):
+ failure_files = sorted(
+ glob.glob(os.path.join(args.inputs_dir, "**", "failures-*.txt"), recursive=True)
+ )
+ ran_files = sorted(
+ glob.glob(os.path.join(args.inputs_dir, "**", "ran-*.txt"), recursive=True)
+ )
+
+ # No per-shard gate lists means the artifacts were never produced or the
+ # download failed (the workflow's download step is continue-on-error). Bail
+ # out before writing an empty baseline-out, which could otherwise be committed
+ # and wipe the entire known-failures.txt.
+ if not failure_files and not ran_files:
+ eprint(
+ "ERROR: no per-shard failures-*.txt / ran-*.txt files found under "
+ "{}. Refusing to aggregate an empty baseline (gate-list artifacts are "
+ "missing or were not downloaded).".format(args.inputs_dir)
+ )
+ return 2
+
+ # Completeness guard: each shard writes its failures-<id>.txt and ran-<id>.txt
+ # together (see run_enforce), so a shard that died before the gate step -- or
+ # whose artifact failed to download -- contributes neither. The empty-inputs
+ # check above only catches losing *all* shards; without this a partial set
+ # would silently regenerate a baseline missing that shard's failures, wrongly
+ # shrinking known-failures.txt and reddening the next run. A shard counts as
+ # complete only when BOTH its files are present (robust to partial downloads).
+ if args.expected_shards:
+ complete = _shard_ids(failure_files, "failures-") & _shard_ids(
+ ran_files, "ran-"
+ )
+ if len(complete) != args.expected_shards:
+ eprint(
+ "ERROR: expected {} shard gate-list set(s) but found {} complete "
+ "(shards: {}). A shard's failures-*/ran-* artifact is missing -- it "
+ "likely died before writing its gate lists or its artifact failed "
+ "to download -- so the regenerated baseline would be incomplete and "
+ "could wrongly shrink known-failures.txt. Refusing to aggregate.".format(
+ args.expected_shards,
+ len(complete),
+ ", ".join(sorted(complete)) or "none",
+ )
+ )
+ return 2
+
+ union_failed = set()
+ for f in failure_files:
+ union_failed |= load_entries(f)
+ union_ran = set()
+ for f in ran_files:
+ union_ran |= load_entries(f)
+
+ flaky_is = make_is_flaky(load_entries(args.flaky_tests))
+ # Exclude quarantined flaky tests from the regenerated baseline: a flaky test
+ # that happened to fail this run must never be baked into known-failures.txt
+ # (otherwise it would trip the now-passing gate on the next run where it
+ # passes). Flaky failures are tracked in flaky-tests.txt, not the baseline.
+ baseline_body = {e for e in union_failed if not flaky_is(e)}
+
+ header = (
+ "# Known Delta-on-Gluten unit test failures.\n"
+ "#\n"
+ "# Auto-generated by compare-test-results.py --mode aggregate.\n"
+ "# Format: <fully.qualified.SuiteName>#<test display name>\n"
+ "# Lines starting with '#' are comments.\n"
+ "#\n"
+ "# Regenerate by running the 'Delta Spark UT (Gluten)' workflow with\n"
+ "# update_baseline=true and committing the produced artifact.\n"
+ )
Review Comment:
Not needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-4979728602
From what I can see in https://github.com/apache/gluten/pull/12292, `TIMESTAMP_NTZ` causes some fallbacks in the query plan, which is expected until this type is fully supported by functions and aggregates. However, the Gluten Delta implementation currently assumes there is always a `WholeStageTransformer`.
It seems that, rather than fixing `TIMESTAMP_NTZ` itself, we may need to make Gluten Delta handle fallbacks correctly. What do you think?
cc @zhztheplayer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3532693217
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,685 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Delta `-DsparkVersion` value (must match the Gluten -P profile below)'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. Spark 4.1 + Scala 2.13 + JDK 17 matches Delta v4.2.0's
+ # default Spark version (4.1.0) from project/CrossSparkVersions.scala.
+ GLUTEN_SPARK_PROFILE: 'spark-4.1'
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: '4.1'
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Every input has a default (workflow_call + workflow_dispatch), so they
+ # are always set; just surface them as step outputs for the steps below.
+ env:
+ DELTA_REF: ${{ inputs.delta_ref }}
+ SPARK_VERSION: ${{ inputs.spark_version }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline }}
+ FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
Review Comment:
spark_version is now the single source of truth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-4989583126
Sure. I’ll open an issue later today.
If the Gluten tests are simply copies from Delta we can remove. I wouldn’t remove immediately, but after some bake time to ensure the new approach is good and stable.
One thing to consider is I’m running Delta 4.2 + Spark 4.1 tests only, the complete matrix would be huge with Delta and Spark releasing several versions per year, for example future Delta 4.4 alone would need 12 shards for Spark 4.0, 4.1 and 4.2, with 4 shards each.
For now I believe Delta 4.2 and Spark 4.1 only are a good start.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3487659897
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,467 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline is empty (not yet bootstrapped) the mode automatically
+ degrades to ``seed`` so the first run is never spuriously red.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard).
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import glob
+import os
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped) sets of (suite, test) tuples. A test is
+ 'failed' if its <testcase> has a <failure> or <error> child, 'skipped' if
+ it has a <skipped> child, otherwise 'passed'. Suite-level aborts (a
+ <testsuite> reporting errors/failures with no failing <testcase>) are
+ recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = (suite, name)
+ tags = _child_local_tags(tc)
+ if "failure" in tags or "error" in tags:
+ failed.add(key)
+ suite_has_failing_tc = True
+ elif "skipped" in tags:
+ skipped.add(key)
+ else:
+ passed.add(key)
+
+ # Suite-level abort: counters say something failed but no testcase
+ # carried the failure (the suite blew up in beforeAll/constructor).
+ # Record a
+ # synthetic entry so the regression is visible.
+ try:
+ errors = int(ts.get("errors", "0") or "0")
+ failures = int(ts.get("failures", "0") or "0")
+ except ValueError:
+ errors = failures = 0
+ if (errors + failures) > 0 and not suite_has_failing_tc:
+ failed.add((suite_name, SUITE_ABORTED))
+
+ if not parsed_any:
+ eprint(
+ "WARNING: no JUnit <testsuite> elements found under {}".format(reports_dir)
+ )
+
+ # A test can't be both passed and failed; failure wins. Skipped only counts
+ # if the test was not otherwise seen (e.g. retried).
+ passed -= failed
+ skipped -= failed
+ skipped -= passed
+ return passed, failed, skipped
+
+
+# --------------------------------------------------------------------------- #
+# Reporting helpers
+# --------------------------------------------------------------------------- #
+def _summary_sink():
+ """Return a writer that mirrors to GITHUB_STEP_SUMMARY when available."""
+ path = os.environ.get("GITHUB_STEP_SUMMARY")
+ handle = open(path, "a", encoding="utf-8") if path else None
+
+ def write(line=""):
+ print(line)
+ if handle:
+ handle.write(line + "\n")
+
+ return write, handle
+
+
+def _print_block(write, title, entries, limit=50):
+ write("")
+ write("### {} ({})".format(title, len(entries)))
+ if not entries:
+ return
+ write("")
+ write("```")
+ for i, (suite, test) in enumerate(sorted(entries)):
+ if i >= limit:
+ write("... and {} more".format(len(entries) - limit))
+ break
+ write(format_entry(suite, test))
+ write("```")
+
+
+# --------------------------------------------------------------------------- #
+# Modes
+# --------------------------------------------------------------------------- #
+def run_enforce(args):
+ baseline = load_entries(args.known_failures)
+ passed, failed, skipped = parse_reports(args.reports_dir)
+
+ # Always emit this shard's artifacts for the aggregation job.
+ if args.failures_out:
+ write_entries(args.failures_out, failed)
+ if args.ran_out:
+ write_entries(args.ran_out, passed | failed)
Review Comment:
This is the same `ran_out`/skipped point as the other thread (discussion_r3487628565), so the rationale there applies: `delta_ref` is pinned, the baseline only ever holds failing tests, and `stale` is informational (it doesn't gate the job), so it can't mislabel anything that matters here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5222839360
> LGTM. Thanks. cc @zhouyuan @zhztheplayer
@zhouyuan @zhztheplayer, do you have any additional concerns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5243929053
Thanks all. The follow-ups are tracked at #12743
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-4910808202
@zhztheplayer @philo-he I have been testing the CI and it is now stable and ready to review.
There is some follow-up uncovered by this CI:
- 810 test failures (previously 963, I patched it on Delta repository, FileSourceScanExec -> FileSourceScanLike)
- #9003 (Old)
- #4963 (Old)
- #12377
- #12387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3667723464
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -101,6 +106,80 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the PR
+ # touches high-signal Delta paths -- the Delta integration code
+ # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's own
+ # files -- or carries the `run-delta-ci` opt-in label. Changes to general
+ # Velox/core/native code can also affect Delta offload but are touched
+ # constantly, so per-PR they skip it; the nightly full run (delta_spark_ut.yml
+ # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage
+ # down. NOTE: the label is read from the event that triggered this run, so add
+ # it before/with a push; labeling an already-finished run needs a new push.
+ delta-changes:
+ runs-on: ubuntu-22.04
+ outputs:
+ run_delta: ${{ steps.filter.outputs.run_delta }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ - name: Detect Delta-relevant changes / opt-in label
+ id: filter
+ env:
+ HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'run-delta-ci') }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
Review Comment:
This job is intended to 'fail open' on non-PR triggers, but the expression `github.event.pull_request.labels.*.name` can be undefined when the triggering event is not `pull_request`, which may cause expression-evaluation errors before the shell `run:` logic can handle missing SHAs. Guard the label expression with an event-name check (e.g., only evaluate `contains(...)` when `github.event_name == 'pull_request'`) and default to `false` otherwise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-4994634214
@zhztheplayer [[VL][Delta] Gluten Delta stats tracker should fall back when the stats plan can't be offloaded (ClassCastException on TIMESTAMP_NTZ)](https://github.com/apache/gluten/issues/12538)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5026316287
> For the maintainers @philo-he @zhztheplayer : this runs on every velox_backend_x86 trigger. Fair reasoning, but with https://github.com/apache/gluten/issues/12288 going on it should be a deliberate choice — every PR, Delta paths only, or nightly + label. Any works for me.
@felipepessoto, we should avoid unnecessary GHA resource consumption, since our usage is still running high. That sometimes means making tradeoffs, particularly for failure cases that are rare. With that in mind, I'd suggest scheduling a nightly job that runs the full test set against the latest main, and narrowing the per-PR Delta tests to the most relevant cases that guard the key functionality. The per-PR job could also be triggered only by changes to the Delta module and a few core-related modules, so PRs that don't touch those paths skip it entirely. This keeps fast, focused signal on every PR while still catching the rarer regressions once a day.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5082787468
There is a new regression:
`Streaming MERGE overflow sourceType: BIGINT, targetType: DECIMAL(7,2) followAnsiEnabled: false, ansiEnabled: true, storeAssignmentPolicy: LEGACY`
@philo-he do you think could this be caused by https://github.com/facebookincubator/velox/pull/18107?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3494090796
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,206 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# Shallow clone the requested tag/branch. Fall back to full clone when the ref is a SHA.
+if ! git clone --depth 1 --branch "$DELTA_REF" https://github.com/delta-io/delta.git "$DELTA_DIR"; then
+ echo "Shallow clone of ref '${DELTA_REF}' failed, falling back to full clone."
+ rm -rf "$DELTA_DIR"
+ git clone https://github.com/delta-io/delta.git "$DELTA_DIR"
+ git -C "$DELTA_DIR" checkout "$DELTA_REF"
+fi
Review Comment:
The fallback path does `rm -rf "$DELTA_DIR"` without validating that `DELTA_DIR` is non-empty and not a dangerous value like `/` or `.`. While CI passes a safe path, this script is also runnable locally and a bad argument would delete arbitrary directories.
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,467 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline is empty (not yet bootstrapped) the mode automatically
+ degrades to ``seed`` so the first run is never spuriously red.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard).
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import glob
+import os
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped) sets of (suite, test) tuples. A test is
+ 'failed' if its <testcase> has a <failure> or <error> child, 'skipped' if
+ it has a <skipped> child, otherwise 'passed'. Suite-level aborts (a
+ <testsuite> reporting errors/failures with no failing <testcase>) are
+ recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = (suite, name)
+ tags = _child_local_tags(tc)
+ if "failure" in tags or "error" in tags:
+ failed.add(key)
+ suite_has_failing_tc = True
+ elif "skipped" in tags:
+ skipped.add(key)
+ else:
+ passed.add(key)
+
+ # Suite-level abort: counters say something failed but no testcase
+ # carried the failure (the suite blew up in beforeAll/constructor).
+ # Record a
+ # synthetic entry so the regression is visible.
+ try:
+ errors = int(ts.get("errors", "0") or "0")
+ failures = int(ts.get("failures", "0") or "0")
+ except ValueError:
+ errors = failures = 0
+ if (errors + failures) > 0 and not suite_has_failing_tc:
+ failed.add((suite_name, SUITE_ABORTED))
+
+ if not parsed_any:
+ eprint(
+ "WARNING: no JUnit <testsuite> elements found under {}".format(reports_dir)
+ )
+
+ # A test can't be both passed and failed; failure wins. Skipped only counts
+ # if the test was not otherwise seen (e.g. retried).
+ passed -= failed
+ skipped -= failed
+ skipped -= passed
+ return passed, failed, skipped
+
+
+# --------------------------------------------------------------------------- #
+# Reporting helpers
+# --------------------------------------------------------------------------- #
+def _summary_sink():
+ """Return a writer that mirrors to GITHUB_STEP_SUMMARY when available."""
+ path = os.environ.get("GITHUB_STEP_SUMMARY")
+ handle = open(path, "a", encoding="utf-8") if path else None
+
+ def write(line=""):
+ print(line)
+ if handle:
+ handle.write(line + "\n")
+
+ return write, handle
+
+
+def _print_block(write, title, entries, limit=50):
+ write("")
+ write("### {} ({})".format(title, len(entries)))
+ if not entries:
+ return
+ write("")
+ write("```")
+ for i, (suite, test) in enumerate(sorted(entries)):
+ if i >= limit:
+ write("... and {} more".format(len(entries) - limit))
+ break
+ write(format_entry(suite, test))
+ write("```")
+
+
+# --------------------------------------------------------------------------- #
+# Modes
+# --------------------------------------------------------------------------- #
+def run_enforce(args):
+ baseline = load_entries(args.known_failures)
+ passed, failed, skipped = parse_reports(args.reports_dir)
+
+ # Always emit this shard's artifacts for the aggregation job.
+ if args.failures_out:
+ write_entries(args.failures_out, failed)
+ if args.ran_out:
+ write_entries(args.ran_out, passed | failed)
+
+ write, handle = _summary_sink()
+ try:
+ seeding = args.mode == "seed" or not baseline
+ if seeding and args.mode != "seed":
+ write(
+ "> NOTE: baseline `{}` is empty -- running in SEED mode "
+ "(no failures will be enforced). Bootstrap the baseline from "
+ "the aggregated artifact, commit it, then enforcement begins.".format(
+ args.known_failures
+ )
+ )
Review Comment:
In `--mode enforce`, a missing/incorrect `--known-failures` path results in an empty baseline and the script silently falls back to seed mode (`seeding = ... or not baseline`). That would make CI go green while not actually enforcing regressions if the baseline file is missing or mis-referenced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3561497850
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,208 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# init + shallow fetch resolves a tag, branch OR commit SHA in a single path
+# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the
+# destructive `rm -rf "$DELTA_DIR"` it required. `--` terminates options so a
+# DELTA_REF starting with `-` can't be misread as a git flag (this script is
+# workflow_dispatch-runnable with a user-supplied ref).
+git init -q "$DELTA_DIR"
+git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git
+git -C "$DELTA_DIR" fetch -q --depth 1 origin -- "$DELTA_REF"
+git -C "$DELTA_DIR" checkout -q FETCH_HEAD
+git -C "$DELTA_DIR" --no-pager log -1 --oneline
+echo "::endgroup::"
+
+echo "::group::Injecting Gluten bundle jar onto the spark project's TEST classpath"
+# The Gluten bundle jar must be on the spark project's TEST runtime classpath
+# (so DeltaSQLCommandTest can load org.apache.gluten.GlutenPlugin by name) but
+# NOT on the COMPILE classpath of `sparkV1`, which is the project that holds
+# Delta's main sources. The bundle's transitive contents include extra symbols
+# under `org.apache.spark.sql` that collide with Delta's main sources -- e.g.
+# MergeOutputGeneration.scala imports both `org.apache.spark.sql._` and
+# `org.apache.spark.sql.delta.ClassicColumnConversions._`, and would then fail
+# with `reference to expression is ambiguous`.
+#
+# sbt auto-scans `<baseDirectory>/lib` via `unmanagedBase`. Two relevant
+# projects in Delta v4.2.0 have a `lib/` baseDirectory:
+# - sparkV1: `project in file("spark")` -> spark/lib
+# - spark : `project in file("spark-unified")` -> spark-unified/lib
+# unmanagedJars are project-scoped (NOT inherited by dependents), so dropping
+# the bundle into spark-unified/lib/ adds it to the unified `spark` project's
+# Compile *and* Test classpaths -- but NOT to sparkV1's. That's exactly what
+# we want:
+# * sparkV1/Compile sees ONLY Delta's regular deps -> Delta main compiles.
+# * spark/Test/fullClasspath sees the bundle -> tests load GlutenPlugin.
+# (Verified empirically: with bundle only in spark-unified/lib/, sbt's
+# `show sparkV1/Compile/dependencyClasspath` excludes the bundle and
+# `show spark/Test/fullClasspath` includes it.)
+#
+# We deliberately do NOT also drop the bundle into spark/lib/, which is what
+# caused the previous compile failure: spark/lib/ is sparkV1's unmanagedBase,
+# and putting the bundle there would re-introduce the ambiguity errors.
+SPARK_UNIFIED_LIB="$DELTA_DIR/spark-unified/lib"
+mkdir -p "$SPARK_UNIFIED_LIB"
+cp "$GLUTEN_BUNDLE_JAR" "$SPARK_UNIFIED_LIB/gluten-velox-bundle.jar"
+ls -lh "$SPARK_UNIFIED_LIB"
+echo "::endgroup::"
+
+echo "::group::Patching DeltaSQLCommandTest to enable Gluten plugin"
+TARGET="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$TARGET" ]; then
+ echo "Expected file not found in Delta clone: $TARGET" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
Review Comment:
Don’t need it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan merged PR #12388:
URL: https://github.com/apache/gluten/pull/12388
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3750480821
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,469 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Self-contained: this workflow owns the whole Delta pipeline, including its own
+ # native build, and is NOT called by velox_backend_x86.yml. A change confined to
+ # this pipeline's own files no longer drags in that workflow's ~50-job
+ # TPC-H/DS + Spark-UT matrix (measured: ~1750 runner-minutes per run that such a
+ # change cannot affect), and a Velox/core change no longer carries the Delta
+ # suite.
+ #
+ # Note the tradeoff: `gluten-delta/**` and `backends-velox/src-delta*/**` also
+ # match velox_backend_x86.yml's own filter (its spark-ut jobs build with
+ # -Pdelta), so a change there runs BOTH workflows and, now that the native lib
+ # is no longer shared between them, pays for the centos-7 build twice (~10 min).
+ # That is the price of decoupling, and it is small next to the case above.
+ #
+ # The `paths:` filter below IS the per-PR gate -- GitHub evaluates it before the
+ # run is created, so a non-Delta PR costs nothing at all. Changes to general
+ # Velox/core/native code can also affect Delta offload, but they are touched on
+ # most PRs; the nightly `schedule` run below is the safety net for those, and
+ # `workflow_dispatch` lets anyone run the suite against a branch on demand.
+ pull_request:
+ paths:
+ - '.github/workflows/delta_spark_ut.yml'
+ - '.github/workflows/util/delta-spark-ut/**'
+ - 'gluten-delta/**'
+ # Covers src-delta, src-delta33, src-delta40 and any future variant.
+ - 'backends-velox/src-delta*/**'
Review Comment:
It is a trade off we made to minimize the CI pipeline usage.
The CI also runs nightly to catch regressions caused by non-Delta changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3750480821
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,469 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Self-contained: this workflow owns the whole Delta pipeline, including its own
+ # native build, and is NOT called by velox_backend_x86.yml. A change confined to
+ # this pipeline's own files no longer drags in that workflow's ~50-job
+ # TPC-H/DS + Spark-UT matrix (measured: ~1750 runner-minutes per run that such a
+ # change cannot affect), and a Velox/core change no longer carries the Delta
+ # suite.
+ #
+ # Note the tradeoff: `gluten-delta/**` and `backends-velox/src-delta*/**` also
+ # match velox_backend_x86.yml's own filter (its spark-ut jobs build with
+ # -Pdelta), so a change there runs BOTH workflows and, now that the native lib
+ # is no longer shared between them, pays for the centos-7 build twice (~10 min).
+ # That is the price of decoupling, and it is small next to the case above.
+ #
+ # The `paths:` filter below IS the per-PR gate -- GitHub evaluates it before the
+ # run is created, so a non-Delta PR costs nothing at all. Changes to general
+ # Velox/core/native code can also affect Delta offload, but they are touched on
+ # most PRs; the nightly `schedule` run below is the safety net for those, and
+ # `workflow_dispatch` lets anyone run the suite against a branch on demand.
+ pull_request:
+ paths:
+ - '.github/workflows/delta_spark_ut.yml'
+ - '.github/workflows/util/delta-spark-ut/**'
+ - 'gluten-delta/**'
+ # Covers src-delta, src-delta33, src-delta40 and any future variant.
+ - 'backends-velox/src-delta*/**'
Review Comment:
It is a trade off we made to minimize the CI pipeline.
The CI also runs nightly to catch regressions caused by non-Delta changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-5135529692
@philo-he, @zhouyuan
#### 1. Delta own workflow - done in this PR
I've split it. `delta_spark_ut.yml` already had its own `build-native-lib-centos-7` for the `workflow_dispatch`/`schedule` path, so it was mostly dropping `workflow_call`, adding a `pull_request` trigger with a Delta `paths:` filter, and deleting the caller-side jobs.
#### 2. Sharing the native lib
Worth doing, but as a follow-up: I measured `build-native-lib-centos-7` at a median of **10.4 min** over the last 16 runs, so it's ~10 min on `gluten-delta` PRs, which is why I didn't block the split on it.
@zhouyuan one wrinkle on the mechanism: artifacts are scoped to a workflow **run**, and there's no cross-workflow `needs`, so the Delta workflow would have to find the sibling run and poll it, both start from the same PR event, so it'd be waiting on a race. A **cache** avoids that entirely since it's populated by push-to-`main`, which is what `velox_backend_cache.yml` already
does for ccache. Your fallback point is right and comes for free: a miss just builds.
Two things to get right first: the key must cover `cpp/**`, the build flags and the base image (a too-narrow key is harmless for ccache, which re-hashes each file itself, but for a *prebuilt* lib it means silently testing a stale binary), and the repo is already at **10.63 GB against GitHub's 10 GB** cache quota, so entries are being evicted, presumably why the other workflows
restore from Stash first.
#### 3. Sharding - done in this PR
Agreed, and it's cheap. A shard is ~9 min fixed setup plus ~135 min of tests, so ~91% shards away: 4 -> 8 shards takes the slowest shard from ~148 min to ~74 min for ~7% more runner-minutes. This also matches delta-io/delta's own `spark_test.yaml`, which runs these suites with `NUM_SHARDS: 8` and `TEST_PARALLELISM_COUNT=4`.
Memory is unchanged (each shard is its own runner), and no baseline regeneration is needed, the gate compares `(suite, test)` sets, so it doesn't care how suites are distributed.
#### 4. Multiple Spark versions
The workflow already takes `spark_version` and `delta_ref` inputs.
The real cost is the **baseline**: `known-failures.txt` currently holds 737 entries specific to Delta 4.2 + Spark 4.1. Every additional combination needs its own baseline file plus a bootstrap run to generate it, and then ongoing triage when it drifts. Each leg is also ~570 runner-minutes.
I wanted to first have the most current version running to prove the approach is right and then expand to more Delta/Spark versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3488621143
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,682 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Delta `-DsparkVersion` value (must match the Gluten -P profile below)'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. Spark 4.1 + Scala 2.13 + JDK 17 matches Delta v4.2.0's
+ # default Spark version (4.1.0) from project/CrossSparkVersions.scala.
+ GLUTEN_SPARK_PROFILE: 'spark-4.1'
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: '4.1'
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Every input has a default (workflow_call + workflow_dispatch), so they
+ # are always set; just surface them as step outputs for the steps below.
+ env:
+ DELTA_REF: ${{ inputs.delta_ref }}
+ SPARK_VERSION: ${{ inputs.spark_version }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline }}
+ FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
+
+ - name: Download Gluten bundle jar
+ uses: actions/download-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: gluten-bundle
+
+ - name: Install minimal tools
+ run: |
+ set -euo pipefail
+ # apache/gluten:centos-9-jdk17 already has java-17, git, tar, a POSIX
+ # shell, and curl-minimal (which provides the `curl` command sbt's
+ # launcher needs). Install the rest of what Delta's build/sbt and the
+ # tests may need. We deliberately do NOT install the full `curl`
+ # package -- it conflicts with the pre-installed curl-minimal.
+ yum install -y java-17-openjdk-devel which findutils gzip python3
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ git --version
+ curl --version | head -n 1
+
+ - name: Cache sbt / Ivy / Coursier
+ uses: actions/cache@v4
+ with:
+ path: |
+ /root/.sbt
+ /root/.ivy2
+ /root/.cache/coursier
+ # Intentionally NOT keyed by ${{ matrix.shard }} -- all shards share
+ # the same dependency tree, so a single shared cache (with parallel
+ # save races resolved by GH on a first-write-wins basis) gives the
+ # best storage / hit-rate tradeoff.
+ key: delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-${{ env.DELTA_SCALA_VERSION }}
+ restore-keys: |
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-
+
+ - name: Clone and patch Delta
+ run: |
+ set -euo pipefail
+ # `2>/dev/null ... || true` keeps a no-match `ls` from aborting the step
+ # under `set -o pipefail`, so the explicit check below emits a clear
+ # error instead of a generic "cannot access".
+ GLUTEN_JAR=$(ls "$GITHUB_WORKSPACE"/gluten-bundle/gluten-velox-bundle-spark*_*-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$GLUTEN_JAR" ] || [ ! -f "$GLUTEN_JAR" ]; then
+ echo "ERROR: No Gluten bundle jar found under $GITHUB_WORKSPACE/gluten-bundle/" >&2
+ ls -la "$GITHUB_WORKSPACE/gluten-bundle/" || true
+ exit 1
+ fi
+ echo "Using Gluten bundle: $GLUTEN_JAR"
+ bash "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/setup-delta.sh" \
+ "${{ steps.resolve.outputs.delta_ref }}" \
+ "$GITHUB_WORKSPACE/delta" \
+ "$GLUTEN_JAR" \
+ "$GITHUB_WORKSPACE"
+
+ - name: Run Delta spark module tests (shard ${{ matrix.shard }} / ${{ env.DELTA_NUM_SHARDS }})
+ env:
+ NUM_SHARDS: ${{ env.DELTA_NUM_SHARDS }}
+ TEST_PARALLELISM_COUNT: ${{ steps.resolve.outputs.test_parallelism }}
+ # Required by Delta to enable testing-only code paths
+ # (see delta build.sbt: "Test / envVars += DELTA_TESTING -> 1").
+ DELTA_TESTING: '1'
+ # JDK 17 + Gluten/Arrow/Netty requires extra --add-opens and the
+ # `io.netty.tryReflectionSetAccessible` system property; otherwise
+ # the forked test JVM fails with
+ # java.lang.UnsupportedOperationException: sun.misc.Unsafe or
+ # java.nio.DirectByteBuffer.<init>(long, int) not available
+ # as soon as Gluten's bundled Arrow allocator initializes Netty
+ # direct buffers. Delta's own `Test / javaOptions` (see
+ # project/CrossSparkVersions.scala `java17TestSettings`) sets the
+ # base add-opens but NOT the Netty property -- Delta's own tests
+ # don't load Arrow/Netty buffers in a way that triggers it.
+ #
+ # Use JAVA_TOOL_OPTIONS so the flags propagate to BOTH the sbt
+ # launcher JVM and the forked test JVM (sbt forks tests and the
+ # child inherits the parent's env). The set below mirrors
+ # `extraJavaTestArgs` from Gluten's own root pom.xml (the
+ # canonical Gluten test JVM flag set).
+ #
+ # NOTE: we deliberately do NOT put `-Xmx` here. JAVA_TOOL_OPTIONS
+ # is processed BEFORE the JVM command line, so Delta's explicit
+ # `-Xmx1024m` (set in build.sbt `Test / javaOptions`) would still
+ # win (last `-Xmx` wins). The forked-test-JVM heap is bumped via
+ # an sbt `set spark / Test / javaOptions ++= ...` command below,
+ # which APPENDS to Delta's own seq -- so our `-Xmx` lands AFTER
+ # `-Xmx1024m` and wins.
+ JAVA_TOOL_OPTIONS: >-
+ -XX:+IgnoreUnrecognizedVMOptions
+ --add-opens=java.base/java.lang=ALL-UNNAMED
+ --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
+ --add-opens=java.base/java.lang.reflect=ALL-UNNAMED
+ --add-opens=java.base/java.io=ALL-UNNAMED
+ --add-opens=java.base/java.net=ALL-UNNAMED
+ --add-opens=java.base/java.nio=ALL-UNNAMED
+ --add-opens=java.base/java.util=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
+ --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.cs=ALL-UNNAMED
+ --add-opens=java.base/sun.security.action=ALL-UNNAMED
+ --add-opens=java.base/sun.util.calendar=ALL-UNNAMED
+ -Djdk.reflect.useDirectMethodHandle=false
+ -Dio.netty.tryReflectionSetAccessible=true
+ -Dfile.encoding=UTF-8
+ run: |
+ set -euo pipefail
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ cd "$GITHUB_WORKSPACE/delta"
+ chmod +x build/sbt
+ # Only run the unified `spark` sbt project, NOT `sparkGroup/test` --
+ # `sparkGroup` aggregates many other projects (sparkV2, contribs,
+ # sharing, connect*, ...) that are out of scope for this pipeline.
+ #
+ # JVM heap layout -- two memory consumers on the ~16G runner:
+ # * sbt launcher JVM: -J-Xmx4G for the test compile, then forced to
+ # return idle memory during the (long) test phase via G1 periodic GC
+ # (G1PeriodicGCInterval=10s; G1PeriodicGCSystemLoadThreshold=0 so the
+ # busy fork doesn't suppress it; full STW via
+ # -G1PeriodicGCInvokesConcurrent) that uncommits to a tight free ratio
Review Comment:
The comment references `-G1PeriodicGCInvokesConcurrent`, but the command line below uses `-J-XX:-G1PeriodicGCInvokesConcurrent` (note the `-XX:` and the negation). Updating the comment avoids confusion when someone tries to tune or reason about the GC behavior.
##########
.github/workflows/util/delta-spark-ut/compare-test-results.py:
##########
@@ -0,0 +1,467 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Gate / seed / aggregate the Delta-on-Gluten unit test results.
+
+Running delta-io/delta's ScalaTest suite against the Gluten Velox bundle
+produces many *expected* failures (Gluten does not yet support every Delta
+code path). To keep the red/green signal meaningful while we fix those
+failures incrementally, we maintain a committed baseline of known failing
+tests (``known-failures.txt``) and compare each CI run against it.
+
+This script has three modes:
+
+``enforce`` (default, per shard)
+ Parse the JUnit XML produced by ``sbt spark/test`` (ScalaTest ``-u``
+ reporter) and compare against the baseline:
+
+ * regression -- a test that FAILED but is NOT in the baseline. These
+ fail the build: a previously-passing test just started failing.
+ * expected -- a test that failed and IS in the baseline. Ignored.
+ * fixed -- a baseline test that now PASSES. By default these also
+ fail the build (``--fail-on-fixed true``) so the baseline stays honest
+ and contributors remove entries as they fix them.
+
+ If the baseline is empty (not yet bootstrapped) the mode automatically
+ degrades to ``seed`` so the first run is never spuriously red.
+
+``seed`` (bootstrap / ``update_baseline``)
+ Never fails. Just writes the current shard's failing tests so the baseline
+ can be (re)generated from a real run.
+
+``aggregate`` (final job)
+ Merge every shard's ``--failures-out`` / ``--ran-out`` file into a single,
+ sorted, ready-to-commit ``known-failures.txt`` and report stale baseline
+ entries (tests no longer present in any shard).
+
+Baseline file format (``known-failures.txt``)::
+
+ # comment lines start with '#'
+ <fully.qualified.SuiteName>#<test display name>
+
+The suite is always a JVM class name (dot-separated, never starts with '#'),
+so a line whose first non-space character is '#' is unambiguously a comment,
+and the FIRST '#' after the suite separates suite from the (possibly
+'#'-containing) test name.
+
+Only the Python standard library is used so the script runs in the bare
+centos image used by the Delta UT pipeline with no ``pip install``.
+"""
+
+import argparse
+import glob
+import os
+import sys
+import xml.etree.ElementTree as ET
+
+# Synthetic "test name" recorded when a whole suite aborts (e.g. beforeAll
+# throws) so that the JUnit XML reports a suite-level error with no per-test
+# <testcase>. Without this, a suite that used to pass but now aborts entirely
+# would record zero failing testcases and the regression would be missed.
+SUITE_ABORTED = "<suite aborted>"
+
+SEP = "#"
+
+
+def eprint(*args, **kwargs):
+ print(*args, file=sys.stderr, **kwargs)
+
+
+# --------------------------------------------------------------------------- #
+# Baseline (known-failures.txt) parsing / formatting
+# --------------------------------------------------------------------------- #
+def format_entry(suite, test):
+ return "{}{}{}".format(suite, SEP, test)
+
+
+def parse_entry(line):
+ """Parse a 'suite#test' line into (suite, test) or return None for blanks/comments."""
+ stripped = line.strip()
+ if not stripped or stripped.startswith("#"):
+ return None
+ idx = stripped.find(SEP)
+ if idx < 0:
+ # No separator: treat the whole line as a suite-level entry.
+ return (stripped, SUITE_ABORTED)
+ return (stripped[:idx], stripped[idx + len(SEP) :])
+
+
+def load_entries(path):
+ """Load a set of (suite, test) tuples from a baseline/shard-list file."""
+ entries = set()
+ if not path or not os.path.exists(path):
+ return entries
+ with open(path, "r", encoding="utf-8") as fh:
+ for line in fh:
+ parsed = parse_entry(line)
+ if parsed is not None:
+ entries.add(parsed)
+ return entries
+
+
+def write_entries(path, entries, header=None):
+ """Write a sorted set of (suite, test) tuples to a file."""
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ if header:
+ for hl in header.splitlines():
+ fh.write(hl.rstrip() + "\n")
+ for suite, test in sorted(entries):
+ # Defensive: collapse any stray newlines so each entry stays on one line.
+ safe_test = test.replace("\r", " ").replace("\n", " ")
+ fh.write(format_entry(suite, safe_test) + "\n")
+
+
+# --------------------------------------------------------------------------- #
+# JUnit XML parsing
+# --------------------------------------------------------------------------- #
+def _iter_testsuites(root):
+ """Yield every <testsuite> element regardless of whether the file root is
+ <testsuites> (wrapper) or a single <testsuite>."""
+ tag = root.tag.split("}")[-1] # strip any namespace
+ if tag == "testsuites":
+ for child in root:
+ if child.tag.split("}")[-1] == "testsuite":
+ yield child
+ elif tag == "testsuite":
+ yield root
+
+
+def _child_local_tags(elem):
+ return {c.tag.split("}")[-1] for c in elem}
+
+
+def parse_reports(reports_dir):
+ """Walk reports_dir for JUnit XML and classify every test.
+
+ Returns (passed, failed, skipped) sets of (suite, test) tuples. A test is
+ 'failed' if its <testcase> has a <failure> or <error> child, 'skipped' if
+ it has a <skipped> child, otherwise 'passed'. Suite-level aborts (a
+ <testsuite> reporting errors/failures with no failing <testcase>) are
+ recorded as a synthetic (suite, SUITE_ABORTED) failure.
+ """
+ passed, failed, skipped = set(), set(), set()
+
+ xml_files = []
+ # ScalaTest's -u reporter and Maven surefire both write `TEST-<suite>.xml`
+ # under a `target/.../*-reports/` dir. Restrict the secondary glob to
+ # `target/` so we never parse Delta's own XML *test resources* (which live
+ # under src/test/resources and are not reports). The <testsuite>-root guard
+ # below is a final safety net.
+ for pattern in ("**/TEST-*.xml", "**/target/**/*.xml"):
+ xml_files.extend(glob.glob(os.path.join(reports_dir, pattern), recursive=True))
+ xml_files = sorted(set(xml_files))
+
+ parsed_any = False
+ for xml_file in xml_files:
+ try:
+ tree = ET.parse(xml_file)
+ except ET.ParseError as exc:
+ eprint("WARNING: could not parse {}: {}".format(xml_file, exc))
+ continue
+ root = tree.getroot()
+ root_tag = root.tag.split("}")[-1]
+ if root_tag not in ("testsuites", "testsuite"):
+ continue # not a JUnit report
+
+ for ts in _iter_testsuites(root):
+ parsed_any = True
+ suite_name = ts.get("name") or ""
+ suite_has_failing_tc = False
+ for tc in ts:
+ if tc.tag.split("}")[-1] != "testcase":
+ continue
+ suite = tc.get("classname") or suite_name
+ name = tc.get("name") or ""
+ key = (suite, name)
+ tags = _child_local_tags(tc)
+ if "failure" in tags or "error" in tags:
+ failed.add(key)
+ suite_has_failing_tc = True
+ elif "skipped" in tags:
+ skipped.add(key)
+ else:
+ passed.add(key)
+
+ # Suite-level abort: counters say something failed but no testcase
+ # carried the failure (the suite blew up in beforeAll/constructor).
+ # Record a
+ # synthetic entry so the regression is visible.
+ try:
+ errors = int(ts.get("errors", "0") or "0")
+ failures = int(ts.get("failures", "0") or "0")
+ except ValueError:
+ errors = failures = 0
+ if (errors + failures) > 0 and not suite_has_failing_tc:
+ failed.add((suite_name, SUITE_ABORTED))
+
+ if not parsed_any:
+ eprint(
+ "WARNING: no JUnit <testsuite> elements found under {}".format(reports_dir)
+ )
+
+ # A test can't be both passed and failed; failure wins. Skipped only counts
+ # if the test was not otherwise seen (e.g. retried).
+ passed -= failed
+ skipped -= failed
+ skipped -= passed
+ return passed, failed, skipped
+
+
+# --------------------------------------------------------------------------- #
+# Reporting helpers
+# --------------------------------------------------------------------------- #
+def _summary_sink():
+ """Return a writer that mirrors to GITHUB_STEP_SUMMARY when available."""
+ path = os.environ.get("GITHUB_STEP_SUMMARY")
+ handle = open(path, "a", encoding="utf-8") if path else None
+
+ def write(line=""):
+ print(line)
+ if handle:
+ handle.write(line + "\n")
+
+ return write, handle
+
+
+def _print_block(write, title, entries, limit=50):
+ write("")
+ write("### {} ({})".format(title, len(entries)))
+ if not entries:
+ return
+ write("")
+ write("```")
+ for i, (suite, test) in enumerate(sorted(entries)):
+ if i >= limit:
+ write("... and {} more".format(len(entries) - limit))
+ break
+ write(format_entry(suite, test))
+ write("```")
+
+
+# --------------------------------------------------------------------------- #
+# Modes
+# --------------------------------------------------------------------------- #
+def run_enforce(args):
+ baseline = load_entries(args.known_failures)
+ passed, failed, skipped = parse_reports(args.reports_dir)
Review Comment:
In `--mode enforce`, a missing/typoed `--known-failures` path silently degrades into seed mode because `load_entries()` returns an empty set when the file doesn’t exist. That can mask regressions by making the gate always pass. Consider treating a non-existent baseline file as a configuration error in enforce mode (while still allowing an existing-but-empty baseline file to seed).
##########
.github/workflows/util/delta-spark-ut/known-failures.txt:
##########
@@ -0,0 +1,830 @@
+# Known Delta-on-Gluten unit test failures.
+#
+# Baseline of delta-io/delta `spark` ScalaTest tests EXPECTED to fail under the
+# Gluten Velox bundle. The Delta Spark UT (Gluten) workflow enforces this list:
+# a failing test NOT listed here is a regression (fails CI); a listed test that
+# now passes should be removed. Format: <fully.qualified.SuiteName>#<test name>.
+# Lines starting with '#' are comments. See README.md in this directory.
+#
+# ---------------------------------------------------------------------------
+# Baseline for the committed 4-shard x 4-fork config. Originally seeded from run
+# 27490052632; refreshed from run 28318129710 (4 x 4) after the FileSourceScanLike
+# test fixes (delta-io/delta #7104 + #7105). 811 known failures total.
Review Comment:
The header hard-codes a total failure count (currently 811). The PR description cites a different count (776), and this number will change as the baseline is refreshed, so leaving it here risks drift and confusion. Consider removing the hard-coded total (or regenerating it automatically) and relying on the aggregate job summary/artifact for the authoritative count.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3533078175
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,208 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
Review Comment:
`DELTA_DIR` is accepted as a positional argument and is used for `git init -q "$DELTA_DIR"` / subsequent `cp` and `sed` operations, but the script never validates that it’s non-empty and not a dangerous value (e.g. `""`, `/`, `.`, `..`). Since this script is intended to be runnable via `workflow_dispatch` (user-supplied args), a bad `delta_dir` can cause the script to operate in an unintended directory (including the current working dir). Add an explicit safety check (and ideally require the directory to be absent or empty) before cloning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3502475240
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -0,0 +1,206 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Prepares a delta-io/delta clone for running its `spark` module tests with the
+# Gluten (Velox) bundle jar on the classpath.
+#
+# Usage:
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>
+#
+# Arguments:
+# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
+# delta_dir - destination directory for the Delta clone
+# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
+# gluten_repo_root - path to the Gluten repository root (used to locate
+# backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+#
+
+set -euo pipefail
+
+if [ "$#" -ne 4 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar> <gluten_repo_root>" >&2
+ exit 1
+fi
+
+DELTA_REF="$1"
+DELTA_DIR="$2"
+GLUTEN_BUNDLE_JAR="$3"
+GLUTEN_ROOT="$4"
+
+if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
+ echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
+ exit 1
+fi
+
+# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
+# rather than maintaining a separate copy. This file is compiled as part of the
+# unified `spark` project's Test scope, which has the Gluten bundle on its
+# classpath (via spark-unified/lib/), so the typed GlutenConfig / VeloxDeltaConfig
+# imports resolve correctly.
+PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
+if [ ! -f "$PATCH_SOURCE" ]; then
+ echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
+ exit 1
+fi
+
+echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
+# init + shallow fetch resolves a tag, branch OR commit SHA in a single path
+# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the
+# destructive `rm -rf "$DELTA_DIR"` it required.
+git init -q "$DELTA_DIR"
+git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git
+git -C "$DELTA_DIR" fetch -q --depth 1 origin "$DELTA_REF"
Review Comment:
Done in 6fe545120. The fetch now passes the ref after -- ( git fetch -q --depth 1 origin -- "$DELTA_REF" ), so a ref starting with - can't be misinterpreted as a git option when the script is run manually via workflow_dispatch .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3549413823
##########
.github/workflows/delta_spark_ut.yml:
##########
@@ -0,0 +1,692 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs Delta Lake's `spark` sbt module unit tests against a Gluten Velox bundle
+# that is built from the source in this repository. The pipeline:
+#
+# 1. Builds the Velox/Gluten native libraries (centos-7 + vcpkg, x86_64).
+# 2. Builds the Gluten Java/Scala jars and assembles the
+# `gluten-velox-bundle-spark<spark>_<scala>-linux_amd64-<version>.jar`
+# fat jar for Spark 4.1 + Scala 2.13 + Java 17 with the Delta profile.
+# 3. Clones delta-io/delta at the requested release tag (default `v4.2.0`),
+# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
+# -- see setup-delta.sh for the unmanagedJars scoping rationale),
+# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
+# and runs `sbt spark/test` sharded across the matrix.
+#
+# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
+# of validating Gluten changes against the latest Delta release.
+
+name: Delta Spark UT (Gluten)
+
+on:
+ # Reusable workflow. velox_backend_x86.yml calls this (gated on Delta-relevant
+ # changes) and passes the native-lib + arrow-jars artifacts it already built,
+ # so the expensive native C++ build is NOT duplicated. Those artifacts live in
+ # the CALLER's run (a called workflow runs as part of the caller run), so the
+ # jobs below download them by name. See velox_backend_x86.yml `delta-spark-ut`.
+ #
+ # NOTE: the `pull_request` trigger was removed so this no longer runs as its own
+ # workflow on PRs (which would double-run the Delta suite). velox_backend_x86.yml
+ # is now the single PR entry point; `workflow_dispatch` keeps manual standalone
+ # runs working (those build the native lib themselves -- see build-native-lib).
+ workflow_call:
+ inputs:
+ native_lib_artifact:
+ description: 'Name of the cpp/build artifact uploaded by the caller'
+ type: string
+ required: true
+ arrow_jars_artifact:
+ description: 'Name of the org.apache.arrow jars artifact uploaded by the caller'
+ type: string
+ required: true
+ delta_ref:
+ type: string
+ required: false
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version driving both the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion.'
+ type: string
+ required: false
+ default: '4.1'
+ test_parallelism:
+ type: string
+ required: false
+ default: '4'
+ update_baseline:
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ type: boolean
+ required: false
+ default: true
+ workflow_dispatch:
+ inputs:
+ delta_ref:
+ description: 'delta-io/delta git ref (tag/branch/SHA) to test against'
+ required: true
+ default: 'v4.2.0'
+ spark_version:
+ description: 'Spark version: drives the Gluten bundle profile (-Pspark-<v>) and Delta -DsparkVersion together. Scala 2.13 + JDK 17 are assumed, so pair a non-4.1 value with a compatible delta_ref.'
+ required: true
+ default: '4.1'
+ test_parallelism:
+ description: 'Forked test JVMs per shard (TEST_PARALLELISM_COUNT)'
+ required: true
+ default: '4'
+ update_baseline:
+ description: 'Seed/refresh the known-failures baseline instead of enforcing it'
+ type: boolean
+ required: false
+ default: false
+ fail_on_fixed:
+ description: 'Fail when a baseline test now passes (keeps the baseline honest)'
+ type: boolean
+ required: false
+ default: true
+
+env:
+ ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
+ MVN_CMD: 'build/mvn -ntp'
+ CCACHE_DIR: "${{ github.workspace }}/.ccache"
+ # Gluten profile / bundle naming for the build-gluten-bundle and
+ # delta-spark-test jobs. `spark_version` is the single source of truth for the
+ # Spark version: it drives the Gluten bundle profile (-Pspark-<v>), the bundle
+ # jar name, and Delta's -DsparkVersion, so the tests always run against a bundle
+ # built for the same Spark version (no separate value to keep in sync). Scala
+ # 2.13 + JDK 17 are pinned -- they match Delta v4.2.0's default Spark 4.1.0 from
+ # project/CrossSparkVersions.scala -- so pair a non-default spark_version with a
+ # compatible delta_ref.
+ GLUTEN_SPARK_PROFILE: spark-${{ inputs.spark_version }}
+ GLUTEN_SCALA_PROFILE: 'scala-2.13'
+ GLUTEN_JAVA_PROFILE: 'java-17'
+ GLUTEN_BUNDLE_SPARK_VERSION: ${{ inputs.spark_version }}
+ GLUTEN_BUNDLE_SCALA_VERSION: '2.13'
+ DELTA_SCALA_VERSION: '2.13.16'
+ # Number of shards in the delta-spark-test matrix. Must equal the length of
+ # the `shard` matrix below.
+ #
+ # 4 shards x TEST_PARALLELISM_COUNT=4 gives ~16-way parallelism packed into 4
+ # runner jobs (4 forks each) rather than 16 single-fork jobs -- fewer concurrent
+ # runners for the same throughput. Sharding is by SUITE; total work
+ # (~1250 shard-minutes) is fixed. Each forked test JVM uses ~4G (2G heap + 2G
+ # off-heap), so 4 forks plus the sbt launcher sit close to the ~16G runner limit;
+ # this fits because the worst memory hog (DeletionVectorsSuite 2B-row) is
+ # force-failed in setup-delta.sh.
+ DELTA_NUM_SHARDS: '4'
+
+# No `concurrency:` here on purpose. As a reusable workflow this runs inside the
+# caller's run, where `github.workflow` resolves to the CALLER's name -- a group
+# keyed on it would collide with the caller's own group and, with
+# cancel-in-progress, could cancel the parent run. The caller's concurrency
+# already governs cancellation. (A standalone workflow_dispatch run just won't
+# auto-cancel, which is fine for infrequent manual runs.)
+
+jobs:
+ build-native-lib-centos-7:
+ # Standalone (workflow_dispatch) only. When called by velox_backend_x86.yml
+ # the caller already built the native lib + arrow jars and passes them as
+ # inputs, so this job is skipped and the duplicate native build is avoided.
+ if: github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Get Ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ restore-keys: |
+ ccache-delta-spark-ut-centos7-release-default
+ ccache-centos7-release-default
+ - name: Build Gluten native libraries
+ run: |
+ docker run -v $GITHUB_WORKSPACE:/work -w /work apache/gluten:vcpkg-centos-7-gcc13 bash -c "
+ set -e
+ yum install tzdata -y
+ df -a
+ cd /work
+ export CCACHE_DIR=/work/.ccache
+ export CCACHE_MAXSIZE=1G
+ mkdir -p /work/.ccache
+ ccache -sz
+ bash dev/ci-velox-buildstatic-centos-7.sh
+ ccache -s
+ mkdir -p /work/.m2/repository/org/apache/arrow/
+ cp -r /root/.m2/repository/org/apache/arrow/* /work/.m2/repository/org/apache/arrow/
+ "
+ - name: Save Ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-delta-spark-ut-centos7-release-default-${{github.sha}}
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-native-lib-centos-7-${{github.sha}}
+ path: ./cpp/build/
+ if-no-files-found: error
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ build-gluten-bundle:
+ needs: build-native-lib-centos-7
+ # Run whether the native lib was built here (dispatch -> success) or provided
+ # by the caller (workflow_call -> build-native-lib-centos-7 skipped).
+ if: ${{ always() && needs.build-native-lib-centos-7.result != 'failure' && needs.build-native-lib-centos-7.result != 'cancelled' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ steps:
+ - uses: actions/checkout@v4
+ - name: Download native artifacts
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.native_lib_artifact || format('delta-spark-ut-native-lib-centos-7-{0}', github.sha) }}
+ path: ./cpp/build/
+ - name: Cache Maven repository
+ uses: actions/cache@v4
+ with:
+ path: /root/.m2/repository
+ key: m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-${{ hashFiles('pom.xml', '**/pom.xml') }}
+ restore-keys: |
+ m2-delta-spark-ut-bundle-${{ env.GLUTEN_SPARK_PROFILE }}-${{ env.GLUTEN_SCALA_PROFILE }}-
+ m2-delta-spark-ut-bundle-
+ - name: Download Arrow jars
+ uses: actions/download-artifact@v4
+ with:
+ name: ${{ inputs.arrow_jars_artifact || format('delta-spark-ut-arrow-jars-centos-7-{0}', github.sha) }}
+ path: /root/.m2/repository/org/apache/arrow/
+ - name: Build Gluten Velox + Delta bundle
+ run: |
+ set -euo pipefail
+ yum install -y java-17-openjdk-devel
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ cd "$GITHUB_WORKSPACE"
+ # `install` (not `package`) so the gluten-delta artifact is in the local
+ # m2 repo before the `package/` shaded jar is built. `Dmaven.compiler.release=17`
+ # overrides any user settings.xml that may pin release=1.8 for Java 17 builds.
+ $MVN_CMD clean install \
+ -P${{ env.GLUTEN_SPARK_PROFILE }} \
+ -P${{ env.GLUTEN_SCALA_PROFILE }} \
+ -P${{ env.GLUTEN_JAVA_PROFILE }} \
+ -Pbackends-velox -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+ - name: Stage bundle jar
+ run: |
+ set -euo pipefail
+ mkdir -p bundle-out
+ # Match the renamed fat jar produced by package/pom.xml's copy-fat-jar
+ # exec. The version part may bump (e.g. 1.7.0-SNAPSHOT -> 1.8.0-SNAPSHOT),
+ # so glob the version suffix. `2>/dev/null ... || true` keeps a no-match
+ # `ls` from aborting the step under `set -o pipefail`, so the explicit
+ # check below runs instead of dying with a generic "cannot access".
+ jar=$(ls package/target/gluten-velox-bundle-spark${{ env.GLUTEN_BUNDLE_SPARK_VERSION }}_${{ env.GLUTEN_BUNDLE_SCALA_VERSION }}-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$jar" ] || [ ! -f "$jar" ]; then
+ echo "ERROR: Could not find Gluten bundle jar under package/target/" >&2
+ ls -la package/target/ || true
+ exit 1
+ fi
+ cp "$jar" bundle-out/
+ ls -lh bundle-out/
+ - uses: actions/upload-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: bundle-out/gluten-velox-bundle-spark*_*-linux_amd64-*.jar
+ if-no-files-found: error
+
+ delta-spark-test:
+ needs: build-gluten-bundle
+ # build-gluten-bundle runs via `if: always()` (its build-native-lib-centos-7 need
+ # is skipped on workflow_call), so this job needs an explicit condition too --
+ # otherwise GitHub's transitive skip propagation, seeing the skipped
+ # build-native-lib-centos-7 ancestor, would skip the whole shard matrix.
+ if: ${{ !cancelled() && needs.build-gluten-bundle.result == 'success' }}
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-9-jdk17
+ # 350-min safety cap. With 4 forks per shard the per-shard suites run
+ # 4-at-a-time, so a shard finishes well under this.
+ timeout-minutes: 350
+ strategy:
+ fail-fast: false
+ matrix:
+ # Length of this list MUST equal env.DELTA_NUM_SHARDS.
+ shard: [0, 1, 2, 3]
+ env:
+ # Mirror Delta's spark_test.yaml env vars used by run-tests.py /
+ # TestParallelization.scala.
+ SHARD_ID: ${{ matrix.shard }}
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Resolve workflow inputs
+ id: resolve
+ # Every input has a default (workflow_call + workflow_dispatch), so they
+ # are always set; just surface them as step outputs for the steps below.
+ env:
+ DELTA_REF: ${{ inputs.delta_ref }}
+ SPARK_VERSION: ${{ inputs.spark_version }}
+ TEST_PARALLELISM: ${{ inputs.test_parallelism }}
+ UPDATE_BASELINE: ${{ inputs.update_baseline }}
+ FAIL_ON_FIXED: ${{ inputs.fail_on_fixed }}
+ run: |
+ set -euo pipefail
+ {
+ echo "delta_ref=${DELTA_REF}"
+ echo "spark_version=${SPARK_VERSION}"
+ echo "test_parallelism=${TEST_PARALLELISM}"
+ echo "update_baseline=${UPDATE_BASELINE}"
+ echo "fail_on_fixed=${FAIL_ON_FIXED}"
+ } | tee -a "$GITHUB_OUTPUT"
+
+ - name: Download Gluten bundle jar
+ uses: actions/download-artifact@v4
+ with:
+ name: delta-spark-ut-gluten-bundle-${{github.sha}}
+ path: gluten-bundle
+
+ - name: Install minimal tools
+ run: |
+ set -euo pipefail
+ # apache/gluten:centos-9-jdk17 already has java-17, git, tar, a POSIX
+ # shell, and curl-minimal (which provides the `curl` command sbt's
+ # launcher needs). Install the rest of what Delta's build/sbt and the
+ # tests may need. We deliberately do NOT install the full `curl`
+ # package -- it conflicts with the pre-installed curl-minimal.
+ yum install -y java-17-openjdk-devel which findutils gzip python3
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
+ export PATH=$JAVA_HOME/bin:$PATH
+ java -version
+ git --version
+ curl --version | head -n 1
+
+ - name: Cache sbt / Ivy / Coursier
+ uses: actions/cache@v4
+ with:
+ path: |
+ /root/.sbt
+ /root/.ivy2
+ /root/.cache/coursier
+ # Intentionally NOT keyed by ${{ matrix.shard }} -- all shards share
+ # the same dependency tree, so a single shared cache (with parallel
+ # save races resolved by GH on a first-write-wins basis) gives the
+ # best storage / hit-rate tradeoff.
+ key: delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-${{ env.DELTA_SCALA_VERSION }}
+ restore-keys: |
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-${{ steps.resolve.outputs.spark_version }}-
+ delta-spark-ut-sbt-${{ steps.resolve.outputs.delta_ref }}-
+
+ - name: Clone and patch Delta
+ run: |
+ set -euo pipefail
+ # `2>/dev/null ... || true` keeps a no-match `ls` from aborting the step
+ # under `set -o pipefail`, so the explicit check below emits a clear
+ # error instead of a generic "cannot access".
+ GLUTEN_JAR=$(ls "$GITHUB_WORKSPACE"/gluten-bundle/gluten-velox-bundle-spark*_*-linux_amd64-*.jar 2>/dev/null | head -n 1 || true)
+ if [ -z "$GLUTEN_JAR" ] || [ ! -f "$GLUTEN_JAR" ]; then
+ echo "ERROR: No Gluten bundle jar found under $GITHUB_WORKSPACE/gluten-bundle/" >&2
+ ls -la "$GITHUB_WORKSPACE/gluten-bundle/" || true
+ exit 1
+ fi
+ echo "Using Gluten bundle: $GLUTEN_JAR"
+ bash "$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/setup-delta.sh" \
+ "${{ steps.resolve.outputs.delta_ref }}" \
+ "$GITHUB_WORKSPACE/delta" \
+ "$GLUTEN_JAR" \
+ "$GITHUB_WORKSPACE"
+
+ - name: Run Delta spark module tests (shard ${{ matrix.shard }} / ${{ env.DELTA_NUM_SHARDS }})
+ env:
+ NUM_SHARDS: ${{ env.DELTA_NUM_SHARDS }}
+ TEST_PARALLELISM_COUNT: ${{ steps.resolve.outputs.test_parallelism }}
+ # Required by Delta to enable testing-only code paths
+ # (see delta build.sbt: "Test / envVars += DELTA_TESTING -> 1").
+ DELTA_TESTING: '1'
+ # JDK 17 + Gluten/Arrow/Netty requires extra --add-opens and the
+ # `io.netty.tryReflectionSetAccessible` system property; otherwise
+ # the forked test JVM fails with
+ # java.lang.UnsupportedOperationException: sun.misc.Unsafe or
+ # java.nio.DirectByteBuffer.<init>(long, int) not available
+ # as soon as Gluten's bundled Arrow allocator initializes Netty
+ # direct buffers. Delta's own `Test / javaOptions` (see
+ # project/CrossSparkVersions.scala `java17TestSettings`) sets the
+ # base add-opens but NOT the Netty property -- Delta's own tests
+ # don't load Arrow/Netty buffers in a way that triggers it.
+ #
+ # Use JAVA_TOOL_OPTIONS so the flags propagate to BOTH the sbt
+ # launcher JVM and the forked test JVM (sbt forks tests and the
+ # child inherits the parent's env). The set below mirrors
+ # `extraJavaTestArgs` from Gluten's own root pom.xml (the
+ # canonical Gluten test JVM flag set).
+ #
+ # NOTE: we deliberately do NOT put `-Xmx` here. JAVA_TOOL_OPTIONS
+ # is processed BEFORE the JVM command line, so Delta's explicit
+ # `-Xmx1024m` (set in build.sbt `Test / javaOptions`) would still
+ # win (last `-Xmx` wins). The forked-test-JVM heap is bumped via
+ # an sbt `set spark / Test / javaOptions ++= ...` command below,
+ # which APPENDS to Delta's own seq -- so our `-Xmx` lands AFTER
+ # `-Xmx1024m` and wins.
+ JAVA_TOOL_OPTIONS: >-
+ -XX:+IgnoreUnrecognizedVMOptions
+ --add-opens=java.base/java.lang=ALL-UNNAMED
+ --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
+ --add-opens=java.base/java.lang.reflect=ALL-UNNAMED
+ --add-opens=java.base/java.io=ALL-UNNAMED
+ --add-opens=java.base/java.net=ALL-UNNAMED
+ --add-opens=java.base/java.nio=ALL-UNNAMED
+ --add-opens=java.base/java.util=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
+ --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
+ --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
+ --add-opens=java.base/sun.nio.cs=ALL-UNNAMED
+ --add-opens=java.base/sun.security.action=ALL-UNNAMED
+ --add-opens=java.base/sun.util.calendar=ALL-UNNAMED
+ -Djdk.reflect.useDirectMethodHandle=false
+ -Dio.netty.tryReflectionSetAccessible=true
+ -Dfile.encoding=UTF-8
Review Comment:
root pom's value is a Maven surefire <argLine> , but Delta runs via its own sbt so it never applies
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add Delta Spark UT pipeline gated against a known-failures baseline [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3549647668
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -93,6 +102,23 @@ jobs:
name: velox-native-lib-centos-7-${{github.sha}}
path: ./cpp/build/
if-no-files-found: error
+ # Consumed by the reusable Delta workflow (delta-spark-ut job).
+ - uses: actions/upload-artifact@v4
+ with:
+ name: velox-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
Review Comment:
Fixed in 1409aec23cab3f317e5b9723fe8381bfe5d4cad8
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -93,6 +102,23 @@ jobs:
name: velox-native-lib-centos-7-${{github.sha}}
path: ./cpp/build/
if-no-files-found: error
+ # Consumed by the reusable Delta workflow (delta-spark-ut job).
+ - uses: actions/upload-artifact@v4
+ with:
+ name: velox-arrow-jars-centos-7-${{github.sha}}
+ path: .m2/repository/org/apache/arrow/
+ if-no-files-found: error
+
+ # Delta Spark UT, run via the reusable workflow so it reuses the native lib +
+ # arrow jars built above instead of duplicating the native build. Not gated on
+ # Delta-only paths: core/velox/substrait/cpp/shims changes can affect Delta
+ # query offload, so this runs on every trigger like the other spark-test jobs.
+ delta-spark-ut:
+ needs: build-native-lib-centos-7
+ uses: ./.github/workflows/delta_spark_ut.yml
+ with:
+ native_lib_artifact: velox-native-lib-centos-7-${{ github.sha }}
+ arrow_jars_artifact: velox-arrow-jars-centos-7-${{ github.sha }}
Review Comment:
Fixed in 1409aec23cab3f317e5b9723fe8381bfe5d4cad8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org