You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "unikdahal (via GitHub)" <gi...@apache.org> on 2026/08/08 23:20:38 UTC

[PR] feat: recognize SparkCopyOnWriteScan as a native Iceberg scan [datafusion-comet]

unikdahal opened a new pull request, #5320:
URL: https://github.com/apache/datafusion-comet/pull/5320

     ## Which issue does this PR close?                                                                                                                                                
                                                                                                                                                                                       
     Closes #5319.
   
   ## Rationale for this change
   
   Iceberg's `MERGE`/`UPDATE`/`DELETE` target tables scan through `SparkCopyOnWriteScan`, which
   extends the same `SparkPartitioningAwareScan` base as the already-supported `SparkBatchQueryScan`
   but wasn't in Comet's recognized scan set, so it always fell back to Spark.
   
   ## What changes are included in this PR?
   
   Add `SparkCopyOnWriteScan` to `IcebergReflection.ICEBERG_SCAN_CLASSES`.
   
   ## How are these changes tested?
   
   verified manually against a real Iceberg REST catalog.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: recognize SparkCopyOnWriteScan as a native Iceberg scan [datafusion-comet]

Posted by "unikdahal (via GitHub)" <gi...@apache.org>.
unikdahal commented on PR #5320:
URL: https://github.com/apache/datafusion-comet/pull/5320#issuecomment-5272815945

   CI failures traced to two issues: 
   
   1. outputPartitioning could force partition planning before the COW dynamic-file-filter subquery resolved ( will be fixing it). 
   
   2.  Nested-struct schema evolution hits a positional (not field-id) cast in iceberg-rust's RecordBatchTransformer, which can misalign/crash on evolved COW tables. The fix exists upstream as apache/iceberg-rust#2647 but isn't merged yet.
   
   This PR depends on  apache/iceberg-rust#2647 merging. Hence changing the PR status as draft for now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: recognize SparkCopyOnWriteScan as a native Iceberg scan [datafusion-comet]

Posted by "github-actions[bot] (via GitHub) via github" <gi...@datafusion.apache.org>.
github-actions[bot] commented on PR #5320:
URL: https://github.com/apache/datafusion-comet/pull/5320#issuecomment-6674388064

   Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: recognize SparkCopyOnWriteScan as a native Iceberg scan [datafusion-comet]

Posted by "github-actions[bot] (via GitHub) via github" <gi...@datafusion.apache.org>.
github-actions[bot] closed pull request #5320:   feat: recognize SparkCopyOnWriteScan as a native Iceberg scan                                                                                                                     
URL: https://github.com/apache/datafusion-comet/pull/5320


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org