You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/05/21 15:43:40 UTC
[PR] chore: drop leftover JVM Parquet helpers and the native_datafusion scan name [datafusion-comet]
andygrove opened a new pull request, #4385:
URL: https://github.com/apache/datafusion-comet/pull/4385
## Which issue does this PR close?
Closes #.
## Rationale for this change
After the JVM Parquet reader paths (`native_comet` and `native_iceberg_compat`) were retired, three Spark-derived helpers were left behind with no production callers. They were originally needed for the JVM vectorized reader to clip Spark request schemas against Parquet file schemas and to build predicate column descriptors; the native Parquet scan does all of that in Rust now.
The same cleanup pass surfaced something else: every reference to `native_datafusion` in code, comments, and test names is now misleading. There is only one scan, so naming it adds no information and makes the code read as if other options still exist.
## What changes are included in this PR?
- Delete three dead files:
- `spark/src/main/scala/org/apache/spark/sql/comet/parquet/CometParquetReadSupport.scala`
- `spark/src/main/scala/org/apache/spark/sql/comet/parquet/CometSparkToParquetSchemaConverter.scala` (only referenced by the file above)
- `spark/src/main/java/org/apache/parquet/filter2/predicate/SparkFilterApi.java` (untouched since the initial PR; no callers)
- Rename `CometScanRule.nativeDataFusionScan` to `nativeScan`.
- Update test names that contained `native_datafusion` to just say `native scan` (5 tests in `ParquetReadV1Suite`, 1 in `CometTaskMetricsSuite`).
- Update comments in `ParquetReadSuite`, `ParquetEncryptionITCase`, `CometNativeScan`, `parquet_exec.rs`, and the bug-triage skill that referred to `native_datafusion` or `native_iceberg_compat`.
Net diff: 10 files, +17 / -767.
## How are these changes tested?
Covered by existing tests; the renames are mechanical and the deleted files have no callers.
- `ParquetReadV1Suite "native scan rejects"`: 5/5 pass (the renamed type-mismatch regression tests).
- `make` and `cargo clippy --all-targets --workspace -- -D warnings` both green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: drop leftover JVM Parquet helpers and the native_datafusion scan name [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #4385:
URL: https://github.com/apache/datafusion-comet/pull/4385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org