You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/02/10 15:00:47 UTC
[PR] Remove prefetch feature, V2 scan dead code, and deprecate BatchReader [datafusion-comet]
andygrove opened a new pull request, #3468:
URL: https://github.com/apache/datafusion-comet/pull/3468
## Summary
- Marks `BatchReader` as `@Deprecated` (since 0.14.0) — kept for Iceberg compatibility via `@IcebergApi`
- Removes all prefetch internals from `BatchReader` (fields, methods, `PrefetchTask` inner class)
- Removes `COMET_SCAN_PREFETCH_ENABLED` and `COMET_SCAN_PREFETCH_THREAD_NUM` configs
- Removes `CometPrefetchThreadPool`
- Deletes `CometParquetPartitionReaderFactory` and `CometParquetScan` (V2 Parquet scan dead code)
- Simplifies `CometScanExec.prepareRDD` to always use `newFileScanRDD`
- Cleans up `EliminateRedundantTransitions` V2 dead code path
- Removes prefetch tests and `BatchReader` benchmark case
- Cleans up `CometParquetScan` references in tests
## Context
`BatchReader` is annotated `@IcebergApi` and must be kept for Iceberg compatibility, but Comet's own production code no longer uses it. The prefetch feature was entirely built on `BatchReader` and is dead code now that the `native_iceberg_compat` path uses `NativeBatchReader`. Comet also no longer accelerates V2 Parquet scans, so `CometParquetScan` and `CometParquetPartitionReaderFactory` are dead code.
## Test plan
- [x] `mvn compile -pl common -am` passes
- [x] `mvn compile -pl spark -am` passes
- [x] `mvn test-compile -pl spark -am` passes
- [x] `mvn test -pl common` passes
- [x] Grep confirms no dangling references to `CometPrefetchThreadPool`, `COMET_SCAN_PREFETCH`, `CometParquetPartitionReaderFactory`, or `CometParquetScan`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Remove all remaining uses of legacy BatchReader from Comet [iceberg] [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #3468:
URL: https://github.com/apache/datafusion-comet/pull/3468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org