You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "malinjawi (via GitHub)" <gi...@apache.org> on 2026/10/20 16:33:08 UTC
[PR] [VL] Stop relocating flatbuffers in the Spark 3.x bundle [gluten]
malinjawi opened a new pull request, #13393:
URL: https://github.com/apache/gluten/pull/13393
## What changes are proposed in this pull request?
Since #13096, the Spark 3.4/3.5 bundle ships Arrow under its original package names, but it still relocates `com.google.flatbuffers`. So the bundled `org.apache.arrow.flatbuf.*` classes extend `org.apache.gluten.shaded.com.google.flatbuffers.Table`, while the classes with the same names in Spark's `arrow-format` jar extend `com.google.flatbuffers.Table`:
```
$ javap -cp gluten-velox-bundle-spark3.5_2.12-*.jar org.apache.arrow.flatbuf.Null
public final class org.apache.arrow.flatbuf.Null extends org.apache.gluten.shaded.com.google.flatbuffers.Table
$ javap -cp arrow-format-12.0.1.jar org.apache.arrow.flatbuf.Null
public final class org.apache.arrow.flatbuf.Null extends com.google.flatbuffers.Table
```
When classpath order puts Spark's `arrow-vector` and `arrow-format` jars on opposite sides of the bundle, `ArrowType` and the `flatbuf` classes come from different copies, and class verification fails:
```
java.lang.VerifyError: Bad type on operand stack
Location:
org/apache/arrow/vector/types/pojo/ArrowType.getTypeForField(Lorg/apache/arrow/flatbuf/Field;)Lorg/apache/arrow/vector/types/pojo/ArrowType; @112: invokevirtual
Reason:
Type 'org/apache/arrow/flatbuf/Null' (current frame, stack[1]) is not assignable to 'com/google/flatbuffers/Table'
```
This is the failure currently hitting the ARM `tpc-test-centos8 (quay.io/centos/centos:stream9, spark-3.5, java-8)` job. gluten-it starts the JVM with `-cp package/target/lib/*`, which picks up jars in directory order, and the failures track the runner image exactly. From 2026-10-18 to 2026-10-20:
- `ubuntu-24.04-arm` 20261012.149.1: 0 of 48 jobs hit it.
- `ubuntu-24.04-arm` 20261018.154.1: 4 of 4 jobs hit it.
The same commit, 9ee941103, [failed on .154.1](https://github.com/apache/gluten/actions/runs/39333463531/job/118338487147) and [passed on .149.1](https://github.com/apache/gluten/actions/runs/39347323279/job/118384087739), with the same JDK and the same jars.
It also affects deployments that copy the bundle into `$SPARK_HOME/jars`. wForget reported the mirror-image error on #13096, on Spark 3.5.4 ([job](https://github.com/wForget/benchmarks-spark-native/actions/runs/38689658927/job/116313644110)).
Changes:
- **`package/pom.xml`:** drop the `com.google.flatbuffers` relocation. The bundled Arrow then references the same flatbuffers names as upstream Arrow. Mixing it with Spark's Arrow becomes ordinary version skew instead of a guaranteed VerifyError.
- **`dev/check-arrow-c-shading.sh`:** add a check that no bundled `org/apache/arrow/**` class references anything under `${gluten.shade.packageName}`. It runs in the existing `verify-arrow-c-shading` step. On the current main bundle it fails for 124 classes.
### Why removing the relocation is safe
The relocation came in with #1030, together with the `org.apache.arrow` relocation, to keep Gluten's Arrow 15 apart from Spark's flatbuffers 1.12.0. That isolation stopped working once `flatbuf` kept its real name in #13096. It also isn't needed:
- **Only Arrow uses flatbuffers in the bundle.** Gluten has no direct dependency on flatbuffers and doesn't reference it in its sources.
- **Arrow 15 and Arrow 12 both link against either flatbuffers version.** Arrow 15 references 33 flatbuffers members, and all of them exist in flatbuffers-java 1.12.0 except `Constants.FLATBUFFERS_23_5_26()`. Likewise, every flatbuffers member Arrow 12 references exists in 23.5.26 except `Constants.FLATBUFFERS_1_12_0()`. Those two constants are called only from the generated `ValidateVersion()` methods, and neither Arrow version ever calls `ValidateVersion()`.
- **Recommended setups behave as before.** With `extraClassPath` the bundle comes first, so the bundled Arrow keeps using the bundled flatbuffers 23.5.26, same as today.
- **Spark 4.x is unchanged.** Its bundle doesn't ship Arrow or flatbuffers (`provided` since #12737).
## How was this patch tested?
- **`./build/mvn verify -pl package -am -DskipTests` on every profile set that CI's install jobs use:** the new check reports `OK org/apache/arrow/** references no gluten-shaded package` on all of them:
- Spark 3.4, 3.5, 4.0 and 4.1 with `-Pbackends-velox`;
- Spark 3.5 with `-Pceleborn`;
- Spark 3.5 with `-Pjava-17 -Pkafka -Pspark-ut`;
- Spark 3.5 and 4.0 with `-Piceberg -Pdelta -Ppaimon -Phudi -Pceleborn -Puniffle`;
- Spark 4.2 with `-Pdelta -Pspark-ut`.
- **Against the bundle from main,** the check reports `FAIL org/apache/arrow/** — 124 class(es) reference gluten-shaded packages: org/apache/gluten/shaded/com/google/flatbuffers`.
- **JVM-only repro on JDK 8:** the Spark 3.5 runtime classpath plus the bundle. I ran all 120 orderings of `arrow-vector-12.0.1`, `arrow-format-12.0.1`, `flatbuffers-java-1.12.0`, `arrow-memory-core-12.0.1` and the bundle. Each run round-trips a `Schema` and an IPC record batch.
- main: 80/120 pass and 40/120 fail with the CI's VerifyError.
- This PR: 120/120 pass.
- Spot checks on JDK 17 give the same result.
The new check only runs where the image has `unzip`, for example `tpc-test-ubuntu-oom`, `tpc-test-ubuntu-randomkill` and `tpc-test-centos8-uniffle`. In images without `unzip`, the script already skips every check. That predates this PR and is left for a follow-up.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: IBM Bob
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Stop relocating flatbuffers in the Spark 3.x bundle [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #13393:
URL: https://github.com/apache/gluten/pull/13393#issuecomment-6314122939
related: https://github.com/apache/gluten/issues/13394
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Stop relocating flatbuffers in the Spark 3.x bundle [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan merged PR #13393:
URL: https://github.com/apache/gluten/pull/13393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org