You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "yaooqinn (via GitHub)" <gi...@apache.org> on 2026/05/27 04:06:18 UTC
[PR] [MINOR][VL] Drop modify_arrow_dataset_scan_option.patch [gluten]
yaooqinn opened a new pull request, #12148:
URL: https://github.com/apache/gluten/pull/12148
### What changes were proposed in this pull request?
Drop `ep/build-velox/src/modify_arrow_dataset_scan_option.patch` and the two
shell call sites that apply / stage it:
- `dev/build-arrow.sh` — `patch -p1 < .../modify_arrow_dataset_scan_option.patch`
- `ep/build-velox/src/get-velox.sh` — `cp` + `git add` that copy the patch
into the Velox source tree
### Why are the changes needed?
The patch added a native CSV / dataset scan-option API
(`CsvFragmentScanOptions::from`, `DeserializeMap`, `mapToExpressionLiteral`, …)
that was originally consumed by Gluten's native Arrow CSV reader path.
That path is gone:
- #11190 fell CSV back to vanilla Spark
- `0658e906f` / `97f463813` / `9ea8290a` / `d206c5e20` removed the JVM-side
callers, the `ArrowUtil` reader path, and the
`spark.gluten.sql.native.arrow.reader.enabled` config
Nothing inside Gluten or Velox now references the symbols the patch
introduces, so it's dead code in the build. This PR is the final cleanup in
that chain.
### How was this patch tested?
Local verification on Ubuntu 24.04 / x86_64:
- `grep` across the repo — 0 callers of `CsvFragmentScanOptions`,
`CsvFragmentScanOptions::from`, `DeserializeMap`, `mapToExpressionLiteral`,
`createNative(... FragmentScanOptions)`, `CsvConvertOptions`,
`testCsvConvertOptions`
- `nm -C` on freshly built `libarrow_dataset.a` (both `java-dist` and
`cpp-jni` outputs) — none of the patch's symbols are present
- `arrow_ep/cpp/src/arrow/dataset/file_csv.cc` on disk is the unmodified
upstream source — the patch was never applied with this change in place
- `dev/buildbundle-veloxbe.sh --enable_vcpkg=ON` → BUILD SUCCESS for all 5
Spark profiles (3.3 / 3.4 / 3.5 / 4.0 / 4.1)
- `spark-shell` on the resulting bundle, reading a CSV file, prints
`GlutenFallbackReporter: Validation failed for plan: Scan csv , due to:
Unsupported file format TextReadFormat` and produces a vanilla
`FileScan csv` physical plan — confirming CSV is fallback-by-design and
never enters the native path the dropped patch fed.
Generated-by: Claude claude-opus-4.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [MINOR][VL] Drop modify_arrow_dataset_scan_option.patch [gluten]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #12148:
URL: https://github.com/apache/gluten/pull/12148#issuecomment-4560942830
Closing per @FelixYBW's note above — keeping the CSV scan scaffolding for the potential future customer use case. Thanks for the context!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [MINOR][VL] Drop modify_arrow_dataset_scan_option.patch [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he merged PR #12148:
URL: https://github.com/apache/gluten/pull/12148
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [MINOR][VL] Drop modify_arrow_dataset_scan_option.patch [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #12148:
URL: https://github.com/apache/gluten/pull/12148#issuecomment-4563274017
Thanks, we are now investigating on Velox CSV reader and it seems also have a gaps vs. Spark CSV reader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [MINOR][VL] Drop modify_arrow_dataset_scan_option.patch [gluten]
Posted by "prestodb-ci (via GitHub)" <gi...@apache.org>.
prestodb-ci commented on PR #12148:
URL: https://github.com/apache/gluten/pull/12148#issuecomment-4653579611
Kicking off rebase job, please wait...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [MINOR][VL] Drop modify_arrow_dataset_scan_option.patch [gluten]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn closed pull request #12148: [MINOR][VL] Drop modify_arrow_dataset_scan_option.patch
URL: https://github.com/apache/gluten/pull/12148
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [MINOR][VL] Drop modify_arrow_dataset_scan_option.patch [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on PR #12148:
URL: https://github.com/apache/gluten/pull/12148#issuecomment-4560905818
If the Arrow-based CSV reader is still needed, can we introduce a build option to control whether these related patches are applied, and skip the Arrow Java build accordingly (use the official Arrow JAR from Maven Central instead)? If only a few users rely on it, perhaps we could disable it by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [MINOR][VL] Drop modify_arrow_dataset_scan_option.patch [gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #12148:
URL: https://github.com/apache/gluten/pull/12148#issuecomment-4650762618
Just noted the csv reader is already deleted in PR https://github.com/apache/gluten/pull/12130
Let's clean up the patch as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [MINOR][VL] Drop modify_arrow_dataset_scan_option.patch [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on PR #12148:
URL: https://github.com/apache/gluten/pull/12148#issuecomment-4551393949
Thanks for the PR. Just to confirm – will we permanently remove the arrow based CSV reader support? cc @jinchengchenghh, @FelixYBW
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [MINOR][VL] Drop modify_arrow_dataset_scan_option.patch [gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #12148:
URL: https://github.com/apache/gluten/pull/12148#issuecomment-4560529902
> Thanks for the PR. Just to confirm – will we permanently remove the arrow based CSV reader support? cc @jinchengchenghh, @FelixYBW
No, we may need to enable it later as it's required by one customer. If we decide to enable it, we will maintain the csv scan.
FYI, @zhouyuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org