You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2026/08/18 12:57:32 UTC
(spark) branch master updated: [SPARK-57411][INFRA] Add a daily build with the JDK codegen backend
This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5fd608b82509 [SPARK-57411][INFRA] Add a daily build with the JDK codegen backend
5fd608b82509 is described below
commit 5fd608b825090e9da3b06107dc0756d114fb815b
Author: YangJie <ya...@baidu.com>
AuthorDate: Tue Aug 18 20:57:17 2026 +0800
[SPARK-57411][INFRA] Add a daily build with the JDK codegen backend
### What changes were proposed in this pull request?
Adds `build_codegen_jdk.yml`, a scheduled workflow that runs the test suite with `spark.sql.codegen.compiler=jdk` every other day. The change has two parts:
- `SPARK_CODEGEN_COMPILER` overrides the `spark.sql.codegen.compiler` default, so the workflow can select the backend through `envs`. This follows the shape `SPARK_ANSI_SQL_MODE` already uses for `spark.sql.ansi.enabled`, and the four TPC-DS steps forward the variable explicitly the same way they forward that one. An empty value counts as unset, for the reason in the testing section.
- The workflow raises three job timeouts to 200, 130 and 140, because javac pushes those jobs past their defaults. The inputs it passes came from #58833, split out of this PR per review feedback and now merged. Numbers are in the testing section.
The workflow keeps only the jobs that compile generated code: build, pyspark, pyspark-pandas, sparkr, tpcds-1g, docker-integration-tests, yarn. Docs, lint, buf and ui are skipped because they never reach `CodeGenerator.compile`. It runs on `*/2` days, matching the cadence of the other heavy scheduled builds; daily would have made a non-default backend roughly twice the compute of the next-heaviest scheduled workflow.
### Why are the changes needed?
The default backend is Janino, so `CodeCompilerSuite` is currently the only suite that exercises `JdkCodeCompiler`. Every other generated unit could take the javac path and none of them is tested on it. The SPARK-57403 umbrella aims at `spark.sql.codegen.compiler=jdk` actually meaning javac, and there is no signal today on whether that holds beyond one suite.
This has already cost us. The four silent-wrong-answer bugs fixed in #57768 (a shadowed field, a hidden static method, an instance method clashing with a static, and an array type) were all found by hand-probing reflection semantics rather than by a failing test. Each sits on a path the encoder, `ObjectExpressions` and `Dataset` suites already cover under Janino, so running those suites on the JDK backend is what turns the next such bug into a test failure instead of a wrong query result.
### Does this PR introduce _any_ user-facing change?
No. The config default is unchanged unless `SPARK_CODEGEN_COMPILER` is set to `janino` or `jdk`, which only the new workflow does.
### How was this patch tested?
Three probe runs of the full suite on the JDK backend, driven by a branch commit that flipped the `build_and_test.yml` default `envs` to `jdk`. That commit is reverted here; it existed only to measure the cost and the failure count.
The last of them, run 31593826890, was green across all 33 jobs: 79201 tests, 0 failures. Every job outside `CodeCompilerSuite` passes on javac.
Durations for the jobs that ran near their limit, javac against Janino:
| job | javac | Janino | old cap | new cap |
| --- | --- | --- | --- | --- |
| `sql - other tests` | 171m | 127m | 150 | 200 |
| `pyspark-sql` and the rest of its matrix entry | 109m | 73m | 120 | 130 |
| `docker-integration-tests` | 116m | 102m | 120 | 140 |
Each new cap is the javac figure plus about 20%. The Janino column is the slowest observed across recent runs of `build_main`, `build_java21`, `build_java25`, `build_non_ansi` and `build_uds`, five to twelve samples depending on the job. The 171m is a single measurement: the first two probe runs cancelled that job at the 150 default, which bounds it from below and says nothing more. Next closest after these three was `sql - slow tests` at 124m against a 150 cap, so it needed no change [...]
On test failures, the first run reported 2 out of 76519, in `StateDataSourceTransformWithStateSuiteCheckpointV2`, on leaked file streams. Neither later run reproduced it, and that suite has failed before on a commit using the Janino default, so it is not a javac problem. Its logs have since expired, so run 31593826890 and its 0 failures is the part of this that can still be re-checked.
Reverting the probe then turned up a bug in the env override itself, which is worth recording because it is the one place this change could have broken other callers. `tpcds-1g` failed with:
```
[INVALID_CONF_VALUE.OUT_OF_RANGE_OF_OPTIONS] The value '' in the config
"spark.sql.codegen.compiler" is invalid. It should be one of 'janino, jdk'.
```
The TPC-DS steps read `${{ fromJSON(inputs.envs).SPARK_CODEGEN_COMPILER }}`, and only this PR's new workflow puts that key in its `envs`. For every other caller the expression resolves to nothing, and the runner exports the variable with an empty value rather than omitting it. `System.getenv` reports it as present with a zero-length string, so `getOrElse` returned `""` and `checkValues` rejected it on the first read of the config. The two existing env-driven confs in `SQLConf` avoid t [...]
The conf now filters the empty string out, which keeps the variable's three states distinct: unset and empty both mean use the default, anything else is validated as before. Probed all of them against a forked JVM: unset gives janino, empty gives janino, `jdk` gives jdk, `JDK` gives jdk (the `transform` still lowercases), and `ecj` is still rejected. The empty case fails without the filter and passes with it.
One coverage gap came out of the same review. The fourth TPC-DS step, "Run TPC-DS queries on collated data", had no `env:` block at all, so `TPCDSCollationQueryTestSuite` was still running on Janino inside this very workflow. It now gets the variable like the other three. That step also still lacks `SPARK_ANSI_SQL_MODE`, which `build_non_ansi.yml` would presumably want; left alone here as a separate question.
Locally, `CodeCompilerSuite` passes (60 tests), along with `catalyst/compile` and `catalyst/scalastyle`. `CodeCompilerSuite` and `CodeGenerationSuite` also pass with the default flipped (`SPARK_CODEGEN_COMPILER=jdk build/sbt "catalyst/testOnly *CodeCompilerSuite *CodeGenerationSuite"`, 97 tests), which confirms no test depends on the default being `janino`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
Closes #57925 from LuciferYang/spark-codegen-jdk-daily.
Authored-by: YangJie <ya...@baidu.com>
Signed-off-by: yangjie01 <ya...@baidu.com>
---
.github/workflows/build_and_test.yml | 5 ++
.github/workflows/build_codegen_jdk.yml | 65 ++++++++++++++++++++++
README.md | 1 +
.../org/apache/spark/sql/internal/SQLConf.scala | 3 +-
4 files changed, 73 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml
index 15b55bf9aa49..78dbfc5c2521 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -1348,6 +1348,7 @@ jobs:
SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 build/sbt "sql/testOnly org.apache.spark.sql.TPCDSQueryTestSuite"
env:
SPARK_ANSI_SQL_MODE: ${{ fromJSON(inputs.envs).SPARK_ANSI_SQL_MODE }}
+ SPARK_CODEGEN_COMPILER: ${{ fromJSON(inputs.envs).SPARK_CODEGEN_COMPILER }}
SPARK_TPCDS_JOIN_CONF: |
spark.sql.autoBroadcastJoinThreshold=-1
spark.sql.join.preferSortMergeJoin=true
@@ -1356,6 +1357,7 @@ jobs:
SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 build/sbt "sql/testOnly org.apache.spark.sql.TPCDSQueryTestSuite"
env:
SPARK_ANSI_SQL_MODE: ${{ fromJSON(inputs.envs).SPARK_ANSI_SQL_MODE }}
+ SPARK_CODEGEN_COMPILER: ${{ fromJSON(inputs.envs).SPARK_CODEGEN_COMPILER }}
SPARK_TPCDS_JOIN_CONF: |
spark.sql.autoBroadcastJoinThreshold=10485760
- name: Run TPC-DS queries (Shuffled hash join)
@@ -1363,12 +1365,15 @@ jobs:
SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 build/sbt "sql/testOnly org.apache.spark.sql.TPCDSQueryTestSuite"
env:
SPARK_ANSI_SQL_MODE: ${{ fromJSON(inputs.envs).SPARK_ANSI_SQL_MODE }}
+ SPARK_CODEGEN_COMPILER: ${{ fromJSON(inputs.envs).SPARK_CODEGEN_COMPILER }}
SPARK_TPCDS_JOIN_CONF: |
spark.sql.autoBroadcastJoinThreshold=-1
spark.sql.join.forceApplyShuffledHashJoin=true
- name: Run TPC-DS queries on collated data
run: |
SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 build/sbt "sql/testOnly org.apache.spark.sql.TPCDSCollationQueryTestSuite"
+ env:
+ SPARK_CODEGEN_COMPILER: ${{ fromJSON(inputs.envs).SPARK_CODEGEN_COMPILER }}
- name: Upload test results to report
if: always()
uses: actions/upload-artifact@v7
diff --git a/.github/workflows/build_codegen_jdk.yml b/.github/workflows/build_codegen_jdk.yml
new file mode 100644
index 000000000000..3df5f94ce484
--- /dev/null
+++ b/.github/workflows/build_codegen_jdk.yml
@@ -0,0 +1,65 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Runs the test suite with `spark.sql.codegen.compiler=jdk`, so that codegen
+# reaching the JDK backend is covered by more than `CodeCompilerSuite`. The
+# default backend is Janino, which leaves the javac path (SPARK-57403) exercised
+# only by its own suite even though every generated unit could take it.
+#
+# javac is slower than Janino, so this runs on its own schedule rather than on
+# every push, skips the jobs that do not compile generated code (docs, lint, buf,
+# ui), and raises the timeout of the three jobs that no longer fit their default.
+name: "Build / Codegen JDK backend (master, Scala 2.13, JDK 17)"
+
+on:
+ schedule:
+ - cron: '37 8 */2 * *'
+ workflow_dispatch:
+
+jobs:
+ run-build:
+ permissions:
+ packages: write
+ name: Run
+ uses: ./.github/workflows/build_and_test.yml
+ if: github.repository == 'apache/spark'
+ with:
+ java: 17
+ branch: master
+ hadoop: hadoop3
+ # See SPARK-57411 for the measurements behind these three.
+ build_timeout_minutes: 200
+ pyspark_timeout_minutes: 130
+ docker_integration_tests_timeout_minutes: 140
+ envs: >-
+ {
+ "PYSPARK_IMAGE_TO_TEST": "python-312",
+ "PYTHON_TO_TEST": "python3.12",
+ "SPARK_CODEGEN_COMPILER": "jdk"
+ }
+ jobs: >-
+ {
+ "build": "true",
+ "pyspark": "true",
+ "pyspark-pandas": "true",
+ "sparkr": "true",
+ "tpcds-1g": "true",
+ "docker-integration-tests": "true",
+ "yarn": "true"
+ }
diff --git a/README.md b/README.md
index a8349b9ba5e7..b57b53688f54 100644
--- a/README.md
+++ b/README.md
@@ -34,6 +34,7 @@ This README file only contains basic setup instructions.
| | [](https://github.com/apache/spark/actions/workflows/build_java21.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_java25.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_non_ansi.yml) |
+| | [](https://github.com/apache/spark/actions/workflows/build_codegen_jdk.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_uds.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_rockdb_as_ui_backend.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven.yml) |
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index a10af15b2822..4deaa0bcf7f2 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -2984,7 +2984,8 @@ object SQLConf {
.stringConf
.transform(_.toLowerCase(Locale.ROOT))
.checkValues(Set("janino", "jdk"))
- .createWithDefault("janino")
+ .createWithDefault(
+ sys.env.get("SPARK_CODEGEN_COMPILER").filter(_.nonEmpty).getOrElse("janino"))
val WHOLESTAGE_HUGE_METHOD_LIMIT = buildConf("spark.sql.codegen.hugeMethodLimit")
.internal()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org