Re: [I] Spark's RewriteDataFiles fails if duplicate files exist in a single live manifest [iceberg]

["rshdhere (via GitHub)" <gi...@apache.org>]

rshdhere commented on issue #17042:
URL: https://github.com/apache/iceberg/issues/17042#issuecomment-6198690683

    Picking this up after a discussion in `#dev` on Slack, where a Kafka Connect user ended up with the same data file committed in two separate append snapshots.
   
   Following @snicholasbarton's earlier suggestion and @RussellSpitzer's comment above, I'd like to make `RewriteDataFiles` fail during planning when a data file path has more than one live entry. 
   
   - If both entries are in the same manifest, the rewrite fails late, at the added records > replaced records check during commit.
    - If they're in separate manifests (the Kafka Connect case), the planner reads the file twice and the rewrite silently commits both copies of the rows into the new files, so the duplication becomes permanent.
   
   The plan is to check in core `BinPackRewriteFilePlanner`, so Spark and Flink both get it. It would throw a `ValidationException` listing the duplicated paths before anything is rewritten. It wouldn't dedupe, because that means picking a sequence number for the user.
   
   Russell, I'll start the dev@ heads-up thread you mentioned before I open a PR. Does that scope sound right, or should `RewriteManifests` fail the same way in the same change?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org