You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "sezruby (via GitHub)" <gi...@apache.org> on 2026/06/05 04:21:41 UTC
[PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
sezruby opened a new pull request, #12244:
URL: https://github.com/apache/gluten/pull/12244
## What changes were proposed in this pull request?
Drop the `15.0.0-gluten` custom artifact coordinate that gluten currently builds via `dev/build-arrow.sh`. Switch every `org.apache.arrow:*` dependency to the vanilla `${arrow.version}` coordinate (15.0.0 for Spark 3.x default; 18.1.0 already in the Spark-4.0/4.1 profiles).
The `arrow-gluten.version` property and the `versions:set -DnewVersion=15.0.0-gluten` step in `build-arrow.sh` are removed. The `modify_arrow_dataset_scan_option.patch` is no longer applied to the Arrow Java build.
## Why this is safe — the patch audit
`build-arrow.sh` previously applied four patches and renamed the resulting jars to `15.0.0-gluten`. Auditing each:
| Patch | Lines | Touches | Status after this PR |
|---|---|---|---|
| `modify_arrow.patch` | 135 | C++ only (CMakeLists, ThirdpartyToolchain, helpers.h, Java pom S3/HDFS) | Still applied; no rename needed (CMake patch survives jar coordinate change) |
| `modify_arrow_dataset_scan_option.patch` | 883 | Adds JVM classes `CsvFragmentScanOptions`, `CsvConvertOptions`, `ConvertUtil`, etc.; C++ `file_csv` / Substrait `expression_internal` / `serde` | **No longer applied to Arrow JVM build.** Every gluten consumer of these JVM classes was deleted by #12130 (Arrow-CSV / Arrow-Dataset JVM code path removal). The C++ portion is still applied via Velox's `CMake/resolve_dependency_modules/arrow/` — `get-velox.sh` continues to copy the patch file into the Velox EP. |
| `cmake-compatibility.patch` | 34 | C++ only (CMake policy version) | Unchanged |
| `support_ibm_power.patch` | 28 | Adds `ppc64le→ppcle_64` arch case to `JniLoader.java` in arrow-c and arrow-dataset | **Still applied**; ppc64le CI builds still need it. The patch only adds a switch case — it doesn't change any public Arrow API and doesn't require a custom artifact coordinate. ppc64le devs still get the patched binary by running `build-arrow.sh`, which now installs vanilla `15.0.0` (overriding the Central jars in their local m2). |
A separate sweep (`grep -rn 'org.apache.arrow.dataset' --include='*.java' --include='*.scala'`) shows that after #12130 the only main-source consumers of `arrow-dataset` are `gluten-arrow/.../ArrowNativeMemoryPool` and `ArrowReservationListener`, which use only the upstream `org.apache.arrow.dataset.jni.{NativeMemoryPool, ReservationListener}` types — no patched classes.
## Effect on contributors
- **x86_64 / aarch64:** `dev/build-arrow.sh` is no longer required to bootstrap the build. All Arrow JVM dependencies resolve from Maven Central. CI / local builds skip the ~hour-long Arrow C++/Java compile.
- **ppc64le:** `dev/build-arrow.sh` is still required (for the patched `arrow-c-data`/`arrow-dataset` JNI binaries with ppc64le arch mapping). The script now installs `arrow-vector:15.0.0` etc. into local m2, overriding Central — same dev-loop as before, just without the rename indirection.
## Effect on shading
Independent of bundling. The bundled gluten-velox-bundle still ships unshaded `org.apache.arrow.*` per #12226. This PR doesn't change which Arrow artifacts end up in the bundle — only their coordinate. The follow-up to actually unbundle Arrow (use Spark's shipped Arrow at runtime) is tracked separately in the discussion under #12226.
## How was this patch tested?
- `mvn dependency:tree -pl gluten-arrow` shows every `org.apache.arrow:*` resolved at vanilla `15.0.0` / `18.1.0` from Central.
- Sweep for stale references: `grep -rn 'arrow-gluten\.version\|15\.0\.0-gluten'` — no matches outside of expected pom diff context.
- Manual CI run pending.
## References
- #12130 — removed the Arrow-CSV / Arrow-Dataset JVM consumers that justified `modify_arrow_dataset_scan_option.patch`
- #12226 — fixed Arrow C-Data shading mismatch (independent; addresses the bundled-jar shading bug)
- Discussion on #12226 — `zhztheplayer` and `FelixYBW` proposed this unbundling direction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4657440865
@sezruby Does lance-spark use arrow-c-data.jar? Did you build locally or download from maven?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4656615288
@sezruby, thank you for the continued efforts.
In the CI workflow, should we remove the Arrow artifact sharing from the build job? Including, but not limited to, the following:
https://github.com/apache/gluten/blob/9d1a5fc61f6783dbd76da7e6c8f306fa25df3056/.github/workflows/velox_backend_x86.yml#L83
https://github.com/apache/gluten/blob/9d1a5fc61f6783dbd76da7e6c8f306fa25df3056/.github/workflows/velox_backend_x86.yml#L98-L102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename and depend on vanilla Apache Arrow [gluten]
Posted by "Jenkins-J (via GitHub)" <gi...@apache.org>.
Jenkins-J commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4663752108
@FelixYBW I've created a PR over in the arrow repository related to version 15 (https://github.com/apache/arrow/pull/50138). Let me know if there are any modifications I should make to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename and depend on vanilla Apache Arrow [gluten]
Posted by "prestodb-ci (via GitHub)" <gi...@apache.org>.
prestodb-ci commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4663847757
Kicking off rebase job, please wait...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4661239682
@sezruby
I see @philo-he [mentioned](https://github.com/apache/gluten/pull/12244#issuecomment-4656615288) there are CI steps copying arrow jars for subsequent CI tests. Can we clean them up in this PR? By doing so I think we don't need a debug commit that I mentioned in my last comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4628119301
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4657624457
> based on the following build guide, it appears that java-jni-manylinux-2014 is used in Arrow's official build.
Looks good, this should avoid the libc incompatibilities.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4650893069
> * **ppc64le** has no native in the Central jar.
@Jenkins-J Can you fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4661160706
@sezruby
> One caveat worth flagging: every gluten CI lane runs builddeps-veloxbe.sh with BUILD_ARROW=OFF and uses the pre-built Arrow baked into the Docker image, so the BUILD_ARROW=ON path (where build-arrow.sh actually executes) is not covered by CI.
Yes, and I think we still need to verify the change on CI. Can you push a debug commit to remove the pre-built Arrow Java jar from local Maven repo, and turn `BUILD_ARROW=ON`, then see if CI can pass?
Once verified, that debug commit can be reverted before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4656779180
> We'd also keep an eye on the glibc compatibility of the official arrow-c-data jar.
@zhztheplayer, based on the following build guide, it appears that java-jni-manylinux-2014 is used in Arrow's official build. Since it is CentOS 7-based and built against glibc 2.17, this should ensure glibc compatibility on higher-version environments.
https://arrow.apache.org/java/main/developers/building.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4657285415
> > We'd also keep an eye on the glibc compatibility of the official arrow-c-data jar.
>
> @zhztheplayer, based on the following build guide, it appears that java-jni-manylinux-2014 is used in Arrow's official build. Since it is CentOS 7-based and built against glibc 2.17, this should ensure glibc compatibility on higher-version environments.
>
> https://arrow.apache.org/java/main/developers/building.html
In Gluten, when we convert velox to Arrow C_data, we should link the one in Velox, not the lib in arrow jar, right? Then we copy the data pointers to arrow jar's and send to JVM.
If any project uses Spark's Arrow C_data, then we should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4655516277
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4657592783
> @sezruby Does lance-spark use arrow-c-data.jar?
Yes it does use `arrow-c-data` https://github.com/lance-format/lance-spark/blob/48ddbd12d5bf28c5d886ed99b0f21a3824ef98b3/pom.xml#L182-L186.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4661197938
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4651050460
> Spark 3.3.1 ships Arrow 7.0.0
> Spark 3.4.4 ships Arrow 11.0.0
> Spark 3.5.5 ships Arrow 15.0.0
> Spark 4.0 / 4.1 ship Arrow 18.x
Per @zhztheplayer, once we completely remove the csv reader, we should be able to use Arrow 7 and Arrow 11.
I re-open the PR: https://github.com/apache/gluten/pull/12148
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4651447106
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "sezruby (via GitHub)" <gi...@apache.org>.
sezruby commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4653030267
@zhztheplayer @philo-he Pushed a follow-up commit (`a9d80cce3`) that early-returns from `build_arrow_java()` when `uname -m != ppc64le`, so x86_64 / aarch64 contributors skip the redundant local install — gluten-arrow resolves arrow-c-data / arrow-dataset from Maven Central on those archs. ppc64le path is byte-for-byte unchanged.
Two CI failures on this run, both unrelated to this PR's diff:
- **`spark-test-spark35-slow`** (2m31s) — failed at container init: `Curl error (7): Couldn't connect to server for http://vault.centos.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml`. CentOS 8 mirror outage.
- **`spark-test-spark41`** (22m10s) — failed in a `SparkScriptTransformationExec` test asserting `some_non_existent_command` produces a `SparkException`. The error stack is from Spark's subprocess machinery, not gluten code; looks like an environment / flake.
Could you re-trigger those two lanes when you get a chance?
One caveat worth flagging: every gluten CI lane runs `builddeps-veloxbe.sh` with `BUILD_ARROW=OFF` and uses the pre-built Arrow baked into the Docker image, so the `BUILD_ARROW=ON` path (where `build-arrow.sh` actually executes) is **not covered by CI**. That means the conditional I added isn't exercised by any lane — it relies on a safe-by-construction argument: early-return on non-ppc64le; ppc64le branch byte-for-byte unchanged from before. Worth keeping in mind for review, and probably worth a separate followup to add at least one CI lane that does run `BUILD_ARROW=ON` end-to-end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename and depend on vanilla Apache Arrow [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he merged PR #12244:
URL: https://github.com/apache/gluten/pull/12244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #12244:
URL: https://github.com/apache/gluten/pull/12244#discussion_r3363683631
##########
dev/build-arrow.sh:
##########
Review Comment:
Do we still need to build Arrow Java locally?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4651061612
> Happy to add a follow-up commit gating build_arrow_java on [[ $(uname -m) == ppc64le ]] so x86_64 / aarch64 users skip ~10 min of redundant work.
@sezruby we can give this a try, thanks.
We'd also keep an eye on the glibc compatibility of the official arrow-c-data jar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4661345065
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Drop 15.0.0-gluten Arrow version rename, depend on vanilla Apache Arrow [gluten]
Posted by "sezruby (via GitHub)" <gi...@apache.org>.
sezruby commented on PR #12244:
URL: https://github.com/apache/gluten/pull/12244#issuecomment-4633245359
> Do we still need to build Arrow Java locally?
Mostly no — Maven Central's `arrow-c-data:15.0.0` jar already ships `libarrow_cdata_jni` for `x86_64/` (Linux/macOS/Windows) and `aarch_64/` (Linux/macOS), so x86_64 / aarch64 contributors no longer need the local Java build after this PR.
The reason it's still wired into `dev/builddeps-veloxbe.sh` unconditionally:
- **ppc64le** has no native in the Central jar. `support_ibm_power.patch` (kept) adds the `ppc64le → ppcle_64` arch case to `JniLoader.java` and the local `mvn install` step bakes a locally-built `libarrow_cdata_jni.so` for ppc64le into the resulting `arrow-c-data:15.0.0` jar in `~/.m2`, overriding Central.
Happy to add a follow-up commit gating `build_arrow_java` on `[[ $(uname -m) == ppc64le ]]` so x86_64 / aarch64 users skip ~10 min of redundant work. Holding off in this PR because there's no ppc64le CI lane to confirm the conditional doesn't break the patched build, and I don't have a qemu setup locally to validate it either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org