[GitHub] [incubator-iceberg] aokolnychyi opened a new issue #316: Provide an API to modify records within files

[GitBox <gi...@apache.org>]

aokolnychyi opened a new issue #316: Provide an API to modify records within files
URL: https://github.com/apache/incubator-iceberg/issues/316
 
 
   We need an API to atomically remove some existing files and add new ones to perform "eager" updates and deletes. While `RewriteFiles` is doing exactly this, it seems to be designed for cases like compaction, where a set of files is replaced but the content does not change. 
   
   `RewriteFiles` cannot be used for updates and deletes because of its logic on retries: it will succeed if all files it tries to remove are still there.
   
   Let's consider an example:
   
   - An update starts when the current snapshot is `s1`.
   - While we are doing the update, there might be an append that commits first. Therefore, `s2` is produced and is marked as the current snapshot.
   - `RewriteFiles` detects the change in metadata and retries.
   - The retry attempt succeeds because the files to be deleted are still there even though the append could add new records that match the update condition. The added matching records won't be taken into account.
   
   We need to either extend/modify the retry logic in an existing operation or provide a new API.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org