You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/01/25 16:38:34 UTC
[PR] [WIP] feat: Add native_iceberg_compat support for V2 Parquet scans [datafusion-comet]
andygrove opened a new pull request, #3272:
URL: https://github.com/apache/datafusion-comet/pull/3272
## Which issue does this PR close?
This PR adds support for `native_iceberg_compat` scan implementation with V2 data sources (BatchScanExec with ParquetScan).
## Rationale for this change
Previously, V2 Parquet scans always used the legacy `BatchReader` (the `native_comet` approach) regardless of the `COMET_NATIVE_SCAN_IMPL` setting. This was inconsistent with V1 scans which respect the scan impl configuration.
This change enables V2 scans to use the DataFusion-based `NativeBatchReader` when `native_iceberg_compat` or `auto` is specified, which is important for deprecating the legacy mutable buffer code.
## What changes are included in this PR?
### New Files
- `CometNativeParquetPartitionReaderFactory`: A V2 partition reader factory that uses `NativeBatchReader` (DataFusion-based Parquet reader)
- `CometNativeParquetScan`: A V2 scan trait that creates the new reader factory
### Behavior
- V2 scans with `auto` or `native_iceberg_compat`: Use `CometNativeParquetScan` (DataFusion-based reader)
- V2 scans with `native_comet`: Use existing `CometParquetScan` (legacy JNI-based BatchReader)
- V2 scans with `native_datafusion`: Fall back to Spark (not yet supported for V2)
## How are these changes tested?
- Updated `CometScanRuleSuite` with tests for V2 scan behavior
- Updated `ParquetReadV2Suite` to verify V2 scans work with the new implementation
- All existing tests pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add V2 scan support for `native_iceberg_compat` [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #3272:
URL: https://github.com/apache/datafusion-comet/pull/3272#issuecomment-3813453142
There is probably a simpler way to do this -
For V1 we choose either `BatchReader` (native_comet) or `NativeBAtchReader` (native_iceberg_compat) when we build a partition reader in `CometParquetFileFormat.CometParquetFileFormat` https://github.com/apache/datafusion-comet/blob/62e8c2a79e4c7b6e443ed948c06fa35ce5cc8ac1/spark/src/main/scala/org/apache/comet/parquet/CometParquetFileFormat.scala#L139
The same logic can be applied for V2 in `CometParquetPartitionReaderFactory.buildCometReader` where we are always creating a `BatchReader` https://github.com/apache/datafusion-comet/blob/62e8c2a79e4c7b6e443ed948c06fa35ce5cc8ac1/spark/src/main/scala/org/apache/comet/parquet/CometParquetPartitionReaderFactory.scala#L130
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add V2 scan support for `native_iceberg_compat` [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed pull request #3272: feat: Add V2 scan support for `native_iceberg_compat`
URL: https://github.com/apache/datafusion-comet/pull/3272
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [WIP] feat: Add native_iceberg_compat support for V2 Parquet scans [datafusion-comet]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3272:
URL: https://github.com/apache/datafusion-comet/pull/3272#issuecomment-3796935116
## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/3272?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `176 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 16.46%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`9b2f8fd`](https://app.codecov.io/gh/apache/datafusion-comet/commit/9b2f8fd40ed74105755aeab75270eceea4e8166b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 880 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/3272?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...uet/CometNativeParquetPartitionReaderFactory.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3272?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Fparquet%2FCometNativeParquetPartitionReaderFactory.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9wYXJxdWV0L0NvbWV0TmF0aXZlUGFycXVldFBhcnRpdGlvblJlYWRlckZhY3Rvcnkuc2NhbGE=) | 0.00% | [79 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3272?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...n/scala/org/apache/comet/rules/CometScanRule.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3272?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Frules%2FCometScanRule.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9ydWxlcy9Db21ldFNjYW5SdWxlLnNjYWxh) | 0.00% | [61 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3272?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../apache/comet/parquet/CometNativeParquetScan.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3272?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Fparquet%2FCometNativeParquetScan.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9wYXJxdWV0L0NvbWV0TmF0aXZlUGFycXVldFNjYW4uc2NhbGE=) | 0.00% | [36 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3272?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #3272 +/- ##
=============================================
- Coverage 56.12% 16.46% -39.67%
+ Complexity 976 400 -576
=============================================
Files 119 177 +58
Lines 11743 16228 +4485
Branches 2251 2680 +429
=============================================
- Hits 6591 2672 -3919
- Misses 4012 13151 +9139
+ Partials 1140 405 -735
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/3272?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org