[PR] fix: [branch-1.1] fall back from native V2 scans when the plan reads input_file_name (#6754) [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new pull request, #7018:
URL: https://github.com/apache/datafusion-comet/pull/7018

   ## Which issue does this PR close?
   
   Closes #6707 on `branch-1.1`. #6754 already closed it on `main`.
   
   ## Rationale for this change
   
   This backports #6754 to `branch-1.1`.
   
   The native Iceberg scan is on by default (`spark.comet.scan.icebergNative.enabled=true`). It does not set `InputFileBlockHolder`, which Iceberg's Spark reader sets as it opens each data file. So `input_file_name()`, `input_file_block_start()` and `input_file_block_length()` over an Iceberg table return `""`, `-1` and `-1` for every row, with no error. The native CSV V2 scan has the same problem, but it is off by default. `transformV1Scan` already falls back from the native Parquet scan when the plan uses these expressions (#3312), and `transformV2Scan` had no such check.
   
   It is a silent wrong-result bug (`priority:critical`, `correctness`) that default settings reach. It is not a 1.1.0 regression, because the native Iceberg scan has had the same gap since it became the default. I am proposing it because of the silent wrong results, in the same way as #6270 and #6700.
   
   ## What changes are included in this PR?
   
   One cherry-pick with `-x`. See #6754 for the details.
   
   `transformV2Scan` now takes the whole plan and declines the scan, with a fallback reason, when the plan reads `input_file_name`, `input_file_block_start` or `input_file_block_length`. The Iceberg and CSV V2 scans then stay in Spark's reader. The `CometScanRule`, `iceberg.md` and `datasources.md` changes are identical to `main`.
   
   One adaptation, recorded in the commit message: `CometCsvNativeReadSuite` on this branch lacks the timestamp and TIME tests that #6754 adds its CSV test next to, so the cherry-pick conflicted there. The resolution keeps only the new `input_file_name` test and adds the `CometCsvNativeScanExec` import it needs.
   
   ## How are these changes tested?
   
   Run locally on this branch with JDK 17, on the default Spark 4.1 profile:
   
   - `CometCsvNativeReadSuite` (4 tests, including the new one) and the `iceberg/input_file_name.sql` fixture in `CometSqlFileTestSuite` pass. Scalastyle and spotless ran in the same build.
   - On Spark 3.5 and Scala 2.12, the sources and tests compile.
   - With the `CometScanRule` change taken out, both new tests fail. The other three pass.
   
   By CI's path routing, which I checked with `dev/ci/compute-changes.py`, this PR runs the all-profiles Linux build, macOS, the Spark SQL jobs for 3.5, 4.0 and 4.1, and the four Iceberg versions, so no `run-*` label is needed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] fix: [branch-1.1] fall back from native V2 scans when the plan reads input_file_name (#6754) [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove merged PR #7018:
URL: https://github.com/apache/datafusion-comet/pull/7018


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org