You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "JkSelf (via GitHub)" <gi...@apache.org> on 2026/10/07 08:47:11 UTC
[PR] Add a test reproducing the Arrow Netty NoSuchFieldError on Spark 3.5 with Netty >= 4.1.118 [gluten]
JkSelf opened a new pull request, #13215:
URL: https://github.com/apache/gluten/pull/13215
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
This adds a test that reproduces the `java.lang.NoSuchFieldError: chunkSize` PySpark hits when
Arrow is enabled (`spark.sql.execution.arrow.pyspark.enabled=true`, e.g. `createDataFrame(pandas_df)`
or `toPandas()`) and Gluten runs on a Spark 3.5 distribution whose Netty was upgraded to 4.1.118 or
later.
Root cause: Spark 3.5 ships `arrow-memory-netty` 12.0.1. Its allocator reads a Netty internal field
(`chunkSize`) that Netty 4.1.118 removed. Standalone allocation checks show:
| arrow-memory-netty | Netty 4.1.96 | Netty >= 4.1.118 (incl. 4.2.x) |
|---|---|---|
| 12.0.1 (Spark 3.5) / 14.0.2 | OK | `NoSuchFieldError: chunkSize` |
| 15.x (Gluten) / 18.1.0 (Spark 4.0) | OK | OK |
An in-process test cannot catch this, because Maven resolves Gluten's `arrow-memory-netty` 15.0.0
with Spark's Netty 4.1.96 for `spark-3.5`, and those two are compatible. `ArrowNettyCompatSuite`
therefore runs a small probe (`new RootAllocator()` plus a few allocations, as Spark's
`ArrowUtils.rootAllocator` does) in a child JVM. That JVM's classpath mirrors a deployment: Gluten's
`arrow-memory-core` first, then Spark 3.5's `arrow-memory-netty` 12.0.1 and a chosen Netty version.
The allocation manager type is left to Arrow's classpath detection. `maven-dependency-plugin` copies
the jars into `gluten-ut/test/target/netty-compat/` during `generate-test-resources`. It is skipped
with `-DskipTests`, and the jars never enter the module's own classpath.
Cases:
- Netty 4.1.96: allocation succeeds. This is the control that shows the harness is sound.
- Netty 4.1.118: fails with `NoSuchFieldError: chunkSize`.
- Netty 4.2.12: fails with `NoSuchFieldError: chunkSize`.
The two failure cases document the current behavior. Once a fix makes Spark's `RootAllocator` work
with new Netty, they should be changed to assert success. The suite cancels itself when Gluten's
Arrow is >= 16 (Spark 4.x builds), where the scenario does not exist.
## How was this patch tested?
The new suite was run locally on JDK 17:
./build/mvn test -pl gluten-ut/test -Pspark-ut,backends-velox,spark-3.5 \
-DwildcardSuites=none -Dsuites=org.apache.gluten.memory.arrow.alloc.ArrowNettyCompatSuite
Result: `Tests: succeeded 3, failed 0, canceled 0`. Running the probe by hand with the same
classpaths gives the outcomes in the table above. The cancel path for Arrow >= 16 has not been run
yet.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-5-5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] Add a test reproducing the Arrow Netty NoSuchFieldError on Spark 3.5 with Netty >= 4.1.118 [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13215:
URL: https://github.com/apache/gluten/pull/13215#discussion_r4204907425
##########
gluten-ut/test/src/test/scala/org/apache/gluten/memory/arrow/alloc/ArrowNettyCompatSuite.scala:
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.memory.arrow.alloc
+
+import org.apache.arrow.memory.RootAllocator
+import org.scalatest.funsuite.AnyFunSuite
+import org.slf4j.Logger
+
+import java.io.File
+import java.nio.charset.StandardCharsets
+import java.util.Arrays
+import java.util.concurrent.TimeUnit
+
+/**
+ * Reproduces the `NoSuchFieldError: chunkSize` that PySpark's Arrow paths (createDataFrame /
+ * toPandas with `spark.sql.execution.arrow.pyspark.enabled`) hit with Gluten on a Spark 3.5
+ * distribution whose Netty was upgraded to 4.1.118 or later.
+ *
+ * Spark 3.5 ships arrow-memory-netty 12.0.1, whose allocator reads a Netty internal field that
+ * Netty 4.1.118 removed. The test classpath has Gluten's newer Arrow and Spark's older Netty, which
+ * are compatible, so the failure cannot show up in-process. Each case therefore runs
+ * [[NettyCompatProbe]] in a child JVM whose classpath mirrors a deployment: Gluten's Arrow first
+ * (as with spark.{driver,executor}.extraClassPath), then Spark's arrow-memory-netty and the chosen
+ * Netty. The allocation manager type is left to Arrow's classpath detection, as in production. The
+ * jars come from maven-dependency-plugin, see this module's pom.xml.
+ *
+ * The cases on new Netty assert the failure, documenting the bug. A fix that makes Spark's
+ * `RootAllocator` work there should turn them into success assertions.
+ */
+class ArrowNettyCompatSuite extends AnyFunSuite {
+ private val compatDir = new File("target/netty-compat")
+ private val spark35ArrowNetty = "arrow-memory-netty-12.0.1"
+ private val timeoutSeconds = 120L
+
+ private case class ProbeResult(exitCode: Int, line: String, output: String)
+
+ test("Spark 3.5 arrow-memory-netty works with Netty 4.1.96") {
+ // Control case: proves the child JVM setup itself is sound.
+ val result = runProbe("netty-4.1.96.Final")
+ assert(result.exitCode == 0, s"Probe failed:\n${result.output}")
+ assert(result.line.contains(" OK "), s"Unexpected probe output:\n${result.output}")
+ }
+
+ test("Spark 3.5 arrow-memory-netty fails with Netty 4.1.118") {
+ assertFailedWithChunkSize(runProbe("netty-4.1.118.Final"))
+ }
+
+ test("Spark 3.5 arrow-memory-netty fails with Netty 4.2.12") {
+ assertFailedWithChunkSize(runProbe("netty-4.2.12.Final"))
+ }
+
+ private def assertFailedWithChunkSize(result: ProbeResult): Unit = {
+ assert(result.exitCode == 1, s"Expected the probe to fail, output:\n${result.output}")
+ assert(
+ result.line.contains("NoSuchFieldError") && result.line.contains("chunkSize"),
+ s"Expected NoSuchFieldError: chunkSize, output:\n${result.output}")
+ }
+
+ private def runProbe(nettyDir: String): ProbeResult = {
+ // Spark 3.5 distributions pair Gluten's bundled arrow-memory-core 15.x with Spark's
+ // arrow-memory-netty 12.0.1. Spark 4.x builds use Spark's own Arrow (>= 18), whose netty
+ // module works with new Netty, so this scenario does not exist there.
+ val arrowMajor = arrowCoreMajorVersion
+ assume(
+ arrowMajor > 0 && arrowMajor < 16,
+ s"Scenario only applies when Gluten bundles Arrow < 16, found $arrowMajor")
+ assert(
+ new File(compatDir, spark35ArrowNetty).isDirectory,
+ s"Missing ${compatDir.getAbsolutePath}; run the build through Maven so that " +
+ "maven-dependency-plugin copies the compat jars"
+ )
+
+ val classpath =
+ // Gluten first: the probe, the Scala library it needs and Gluten's arrow-memory-core.
+ Seq(
+ codeSource(NettyCompatProbe.getClass),
+ codeSource(classOf[Option[_]]),
+ codeSource(classOf[RootAllocator])) ++
+ // Then what the Spark distribution provides.
+ jarsIn(new File(compatDir, spark35ArrowNetty)) ++
+ jarsIn(new File(compatDir, nettyDir)) :+
+ codeSource(classOf[Logger])
+
+ val command = Seq(
+ new File(System.getProperty("java.home"), "bin/java").getPath,
+ "--add-opens=java.base/java.nio=ALL-UNNAMED",
Review Comment:
Spark 3.5 is supported on Java 8, but Java 8 rejects `--add-opens` before this probe can start. The root test JVM flags handle this by placing `-XX:+IgnoreUnrecognizedVMOptions` before the module options (`pom.xml:178-185`); do the same here so the suite works under the repository's Java 8 profile.
##########
gluten-ut/test/src/test/scala/org/apache/gluten/memory/arrow/alloc/ArrowNettyCompatSuite.scala:
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.memory.arrow.alloc
+
+import org.apache.arrow.memory.RootAllocator
+import org.scalatest.funsuite.AnyFunSuite
+import org.slf4j.Logger
+
+import java.io.File
+import java.nio.charset.StandardCharsets
+import java.util.Arrays
+import java.util.concurrent.TimeUnit
+
+/**
+ * Reproduces the `NoSuchFieldError: chunkSize` that PySpark's Arrow paths (createDataFrame /
+ * toPandas with `spark.sql.execution.arrow.pyspark.enabled`) hit with Gluten on a Spark 3.5
+ * distribution whose Netty was upgraded to 4.1.118 or later.
+ *
+ * Spark 3.5 ships arrow-memory-netty 12.0.1, whose allocator reads a Netty internal field that
+ * Netty 4.1.118 removed. The test classpath has Gluten's newer Arrow and Spark's older Netty, which
+ * are compatible, so the failure cannot show up in-process. Each case therefore runs
+ * [[NettyCompatProbe]] in a child JVM whose classpath mirrors a deployment: Gluten's Arrow first
+ * (as with spark.{driver,executor}.extraClassPath), then Spark's arrow-memory-netty and the chosen
+ * Netty. The allocation manager type is left to Arrow's classpath detection, as in production. The
+ * jars come from maven-dependency-plugin, see this module's pom.xml.
+ *
+ * The cases on new Netty assert the failure, documenting the bug. A fix that makes Spark's
+ * `RootAllocator` work there should turn them into success assertions.
+ */
+class ArrowNettyCompatSuite extends AnyFunSuite {
+ private val compatDir = new File("target/netty-compat")
+ private val spark35ArrowNetty = "arrow-memory-netty-12.0.1"
+ private val timeoutSeconds = 120L
+
+ private case class ProbeResult(exitCode: Int, line: String, output: String)
+
+ test("Spark 3.5 arrow-memory-netty works with Netty 4.1.96") {
+ // Control case: proves the child JVM setup itself is sound.
+ val result = runProbe("netty-4.1.96.Final")
+ assert(result.exitCode == 0, s"Probe failed:\n${result.output}")
+ assert(result.line.contains(" OK "), s"Unexpected probe output:\n${result.output}")
+ }
+
+ test("Spark 3.5 arrow-memory-netty fails with Netty 4.1.118") {
+ assertFailedWithChunkSize(runProbe("netty-4.1.118.Final"))
+ }
+
+ test("Spark 3.5 arrow-memory-netty fails with Netty 4.2.12") {
+ assertFailedWithChunkSize(runProbe("netty-4.2.12.Final"))
+ }
+
+ private def assertFailedWithChunkSize(result: ProbeResult): Unit = {
+ assert(result.exitCode == 1, s"Expected the probe to fail, output:\n${result.output}")
+ assert(
+ result.line.contains("NoSuchFieldError") && result.line.contains("chunkSize"),
+ s"Expected NoSuchFieldError: chunkSize, output:\n${result.output}")
+ }
+
+ private def runProbe(nettyDir: String): ProbeResult = {
+ // Spark 3.5 distributions pair Gluten's bundled arrow-memory-core 15.x with Spark's
+ // arrow-memory-netty 12.0.1. Spark 4.x builds use Spark's own Arrow (>= 18), whose netty
+ // module works with new Netty, so this scenario does not exist there.
+ val arrowMajor = arrowCoreMajorVersion
+ assume(
+ arrowMajor > 0 && arrowMajor < 16,
+ s"Scenario only applies when Gluten bundles Arrow < 16, found $arrowMajor")
+ assert(
+ new File(compatDir, spark35ArrowNetty).isDirectory,
+ s"Missing ${compatDir.getAbsolutePath}; run the build through Maven so that " +
+ "maven-dependency-plugin copies the compat jars"
+ )
+
+ val classpath =
+ // Gluten first: the probe, the Scala library it needs and Gluten's arrow-memory-core.
+ Seq(
+ codeSource(NettyCompatProbe.getClass),
+ codeSource(classOf[Option[_]]),
+ codeSource(classOf[RootAllocator])) ++
+ // Then what the Spark distribution provides.
+ jarsIn(new File(compatDir, spark35ArrowNetty)) ++
+ jarsIn(new File(compatDir, nettyDir)) :+
+ codeSource(classOf[Logger])
+
+ val command = Seq(
+ new File(System.getProperty("java.home"), "bin/java").getPath,
+ "--add-opens=java.base/java.nio=ALL-UNNAMED",
+ "-cp",
+ classpath.mkString(File.pathSeparator),
+ NettyCompatProbe.getClass.getName.stripSuffix("$")
+ )
+ val builder = new ProcessBuilder(Arrays.asList(command: _*)).redirectErrorStream(true)
+ // Leave the allocation manager type to classpath detection, as in production.
+ builder.environment().remove("ARROW_ALLOCATION_MANAGER_TYPE")
+ val process = builder.start()
+ val output = new String(readAll(process), StandardCharsets.UTF_8)
+ assert(
+ process.waitFor(timeoutSeconds, TimeUnit.SECONDS),
+ s"Probe did not finish within ${timeoutSeconds}s")
+ val line =
+ output.split("\\R").find(_.startsWith(NettyCompatProbe.ResultPrefix)).getOrElse("")
+ ProbeResult(process.exitValue(), line, s"Classpath: $classpath\n$output")
+ }
+
+ private def readAll(process: Process): Array[Byte] = {
+ val in = process.getInputStream
+ try {
+ Iterator.continually(in.read()).takeWhile(_ != -1).map(_.toByte).toArray
+ } finally {
+ in.close()
+ }
+ }
Review Comment:
`readAll` blocks until the child closes stdout, so the timed `waitFor` is not reached until after the child has already exited. If allocation hangs, this test hangs indefinitely instead of enforcing 120 seconds; redirect/drain output independently, kill on timeout, and only then collect the output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] Add a test reproducing the Arrow Netty NoSuchFieldError on Spark 3.5 with Netty >= 4.1.118 [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13215:
URL: https://github.com/apache/gluten/pull/13215#issuecomment-6034369195
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] Add a test reproducing the Arrow Netty NoSuchFieldError on Spark 3.5 with Netty >= 4.1.118 [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13215:
URL: https://github.com/apache/gluten/pull/13215#discussion_r4260070853
##########
.github/workflows/util/pyspark_arrow_test.py:
##########
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs PySpark's Arrow conversions (createDataFrame(pandas_df) and toPandas()) with the Gluten
+# bundle on the classpath, to check that Spark's own Arrow code still works next to the Arrow
+# classes Gluten bundles. Submit it with the bundle in spark.{driver,executor}.extraClassPath and
+# spark.plugins=org.apache.gluten.GlutenPlugin, see the pyspark-arrow-test job in
+# velox_backend_x86.yml.
+
+import numpy as np
+import pandas as pd
+from pyspark.sql import SparkSession
+
+spark = SparkSession.builder.appName("pyspark-arrow-test").getOrCreate()
+assert "org.apache.gluten.GlutenPlugin" in spark.conf.get("spark.plugins", ""), \
+ "Gluten is not enabled, submit with spark.plugins=org.apache.gluten.GlutenPlugin"
Review Comment:
Using `assert` for runtime validation is fragile because asserts can be disabled with Python optimization flags (`-O`), which would make this test silently pass even if Gluten/Arrow behavior is broken. Prefer explicit checks that raise `AssertionError` (or `RuntimeError`) unconditionally so the CI job always fails on unexpected behavior.
##########
.github/workflows/util/pyspark_arrow_test.py:
##########
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs PySpark's Arrow conversions (createDataFrame(pandas_df) and toPandas()) with the Gluten
+# bundle on the classpath, to check that Spark's own Arrow code still works next to the Arrow
+# classes Gluten bundles. Submit it with the bundle in spark.{driver,executor}.extraClassPath and
+# spark.plugins=org.apache.gluten.GlutenPlugin, see the pyspark-arrow-test job in
+# velox_backend_x86.yml.
+
+import numpy as np
+import pandas as pd
+from pyspark.sql import SparkSession
+
+spark = SparkSession.builder.appName("pyspark-arrow-test").getOrCreate()
+assert "org.apache.gluten.GlutenPlugin" in spark.conf.get("spark.plugins", ""), \
+ "Gluten is not enabled, submit with spark.plugins=org.apache.gluten.GlutenPlugin"
+
+spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")
+spark.conf.set("spark.sql.execution.arrow.pyspark.fallback.enabled", "false")
+
+pdf = pd.DataFrame(np.full((100, 3), 5))
+
+df = spark.createDataFrame(pdf)
+result_pdf = df.select("*").toPandas()
+print("Pandas DataFrame result statics:\n%s\n" % str(result_pdf.describe()))
Review Comment:
Typo in log message: 'statics' should be 'statistics'.
##########
.github/workflows/util/pyspark_arrow_test.py:
##########
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs PySpark's Arrow conversions (createDataFrame(pandas_df) and toPandas()) with the Gluten
+# bundle on the classpath, to check that Spark's own Arrow code still works next to the Arrow
+# classes Gluten bundles. Submit it with the bundle in spark.{driver,executor}.extraClassPath and
+# spark.plugins=org.apache.gluten.GlutenPlugin, see the pyspark-arrow-test job in
+# velox_backend_x86.yml.
+
+import numpy as np
+import pandas as pd
+from pyspark.sql import SparkSession
+
+spark = SparkSession.builder.appName("pyspark-arrow-test").getOrCreate()
+assert "org.apache.gluten.GlutenPlugin" in spark.conf.get("spark.plugins", ""), \
+ "Gluten is not enabled, submit with spark.plugins=org.apache.gluten.GlutenPlugin"
+
+spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")
+spark.conf.set("spark.sql.execution.arrow.pyspark.fallback.enabled", "false")
+
+pdf = pd.DataFrame(np.full((100, 3), 5))
+
+df = spark.createDataFrame(pdf)
+result_pdf = df.select("*").toPandas()
+print("Pandas DataFrame result statics:\n%s\n" % str(result_pdf.describe()))
+assert result_pdf.shape == (100, 3), "Unexpected shape %s" % str(result_pdf.shape)
+assert (result_pdf.to_numpy() == 5).all(), "Unexpected values in the result"
Review Comment:
Using `assert` for runtime validation is fragile because asserts can be disabled with Python optimization flags (`-O`), which would make this test silently pass even if Gluten/Arrow behavior is broken. Prefer explicit checks that raise `AssertionError` (or `RuntimeError`) unconditionally so the CI job always fails on unexpected behavior.
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -927,6 +927,48 @@ jobs:
name: spark-test-spark35-group${{ matrix.group }}-golden-files
path: /tmp/tpch-approved-plan/**
+ pyspark-arrow-test-spark35:
+ # PySpark's Arrow conversions run Spark's own Arrow code, which breaks if the Arrow classes
+ # in the Gluten bundle no longer link with it (e.g. the NoSuchMethodError fixed by #13096).
+ needs: [detect-changes, build-native-lib-centos-8]
+ if: >-
+ needs.detect-changes.outputs.java == 'true' ||
+ needs.detect-changes.outputs.shims35 == 'true' ||
+ needs.detect-changes.outputs.cpp == 'true'
+ runs-on: ubuntu-22.04
+ container: apache/gluten:centos-8-jdk8
+ steps:
+ - uses: actions/checkout@v7
+ - name: Download All Artifacts
+ uses: actions/download-artifact@v8
+ with:
+ name: velox-native-lib-centos-8-${{github.sha}}
+ path: ./cpp/build/releases/
+ - name: Prepare
+ run: |
+ dnf module -y install python39 && \
+ alternatives --set python3 /usr/bin/python3.9 && \
+ pip3 install setuptools==77.0.3 && \
+ pip3 install pyspark==3.5.5 pandas==2.2.3 pyarrow==20.0.0
+ - name: Build Gluten bundle
+ run: |
+ cd $GITHUB_WORKSPACE/
+ $MVN_CMD clean package -Pspark-3.5 -Pjava-8 -Pbackends-velox -DskipTests
+ - name: Run PySpark Arrow conversions with the Gluten bundle
+ run: |
+ cd $GITHUB_WORKSPACE/
+ # The pip pyspark package is a full Spark 3.5.5 distribution.
+ export SPARK_HOME=$(python3 -c 'import os, pyspark; print(os.path.dirname(pyspark.__file__))')
+ BUNDLE=$(ls package/target/gluten-velox-bundle-spark3.5_2.12-*.jar)
+ $SPARK_HOME/bin/spark-submit --master 'local[2]' \
+ --conf spark.driver.extraClassPath=$BUNDLE \
+ --conf spark.executor.extraClassPath=$BUNDLE \
Review Comment:
`BUNDLE=$(ls ...)` will fail if the glob matches 0 files, and will produce an invalid classpath if it matches more than 1 file (newline-separated). Also, the `extraClassPath` values are unquoted, which can break if the resolved path ever contains shell-special characters. Consider selecting exactly one jar deterministically (and erroring with a clear message otherwise) and quoting `$BUNDLE` in the `--conf` arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org