[I] Make native Iceberg write coverage visible in the Iceberg Spark test CI jobs [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #6148:
URL: https://github.com/apache/datafusion-comet/issues/6148

   ### What is the problem the feature request solves?
   
   Since #5677, the Iceberg Spark test diffs run the upstream Iceberg suites with the native Iceberg writer on. A green run cannot tell a native write from a silent fallback:
   
   - `CometIcebergNativeWrite` has about 22 eligibility rules that fall back silently, and writes whose input is a local relation fall back unless `spark.comet.exec.localTableScan.enabled` is set.
   - The diffs set `spark.comet.explainFallback.enabled=true`, but gradle does not forward Spark's log output into the CI job log, so the fallback reasons are not visible there.
   - No upstream Iceberg test asserts which writer ran.
   
   So the Iceberg jobs are weak evidence for the "diffs pass with the flags on" criterion in #5644. Comet's own suites do assert `CometIcebergWriteExec` is in the plan, but they run only on Spark 4.1 / Iceberg 1.11 per PR.
   
   ### Describe the potential solution
   
   Make native-write coverage measurable in the Iceberg jobs, for example:
   
   - a listener installed by the diffs that counts `CometIcebergWriteExec` vs `IcebergWriteExec` executions and fallback reasons, and writes them to a file the job uploads or prints as a summary, or
   - forward the `CometExecRule` fallback warnings through gradle's test logging so they can be grepped, or
   - a strict mode (test-only config) that fails a write which falls back for any reason outside an expected list.
   
   A per-version summary (native writes / total writes, top fallback reasons) would also show how much of the suite exercises the native path.
   
   ### Additional context
   
   Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649. Related: #5644.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Make native Iceberg write coverage visible in the Iceberg Spark test CI jobs [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove closed issue #6148: Make native Iceberg write coverage visible in the Iceberg Spark test CI jobs
URL: https://github.com/apache/datafusion-comet/issues/6148


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org