You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mahdibh (via GitHub)" <gi...@apache.org> on 2023/09/25 16:15:38 UTC

[GitHub] [iceberg] mahdibh opened a new issue, #8642: Cannot specify custom delete handler when using a DeleteFiles in a Transaction

mahdibh opened a new issue, #8642:
URL: https://github.com/apache/iceberg/issues/8642

   ### Apache Iceberg version
   
   1.2.0
   
   ### Query engine
   
   None
   
   ### Please describe the bug 🐞
   
   I can specify my own delete handler when using a DeleteFiles operation directly (through `table.newDelete()`). However, when doing it through a transaction (ie, `table.newTransaction().newDelete()`) it fails with the following exception
   ```
    java.lang.IllegalArgumentException: Cannot set delete callback more than once
   	at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
   	at org.apache.iceberg.SnapshotProducer.deleteWith(SnapshotProducer.java:164)
   	at IcebergDeleteTest.main(IcebergDeleteTest.java:42)
   ```
   I would expect the delete to work the same way within a transaction. If the internal code needs to do cleanups, those should be handled separately from the actual cleanups that happen as a result of the delete operation.
   
   Here is a full repro with the successful case first, then the failed one
   
   ```
   import org.apache.hadoop.conf.Configuration;
   import org.apache.iceberg.DeleteFiles;
   import org.apache.iceberg.Schema;
   import org.apache.iceberg.Table;
   import org.apache.iceberg.Transaction;
   import org.apache.iceberg.catalog.Catalog;
   import org.apache.iceberg.catalog.TableIdentifier;
   import org.apache.iceberg.hadoop.HadoopCatalog;
   import org.apache.iceberg.types.Types;
   
   public class IcebergDeleteTest {
     public static void main(String[] args) {
       // Specify the local directory where the Hadoop catalog will store metadata
       String catalogPath = "file:///tmp/catalog";
   
       // Create a Hadoop Catalog
       Catalog catalog = new HadoopCatalog(new Configuration(), catalogPath);
   
       // Define the table identifier
       TableIdentifier tableIdentifier = TableIdentifier.of("my_database", "my_table");
   
       // Create a schema for your data
       Schema schema = new Schema(Types.NestedField.optional(1, "id", Types.IntegerType.get()));
   
       // Create a table
       Table table = catalog.createTable(tableIdentifier, schema);
   
       // delete directly (works fine)
       DeleteFiles deleteFiles = table.newDelete();
       deleteFiles
           .deleteWith(
               (path) -> {
                 System.err.println("deleting file " + path);
               })
           .deleteFile("file://foo1")
           .commit();
   
       // delete through a transaction (throws exception)
       Transaction transaction = table.newTransaction();
       deleteFiles = transaction.newDelete();
       deleteFiles
           .deleteWith(
               (path) -> {
                 System.err.println("deleting file " + path);
               })
           .deleteFile("file://foo2")
           .commit();
   
       transaction.commitTransaction();
     }
   }
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] mahdibh commented on issue #8642: Cannot specify custom delete handler when using a DeleteFiles in a Transaction

Posted by "mahdibh (via GitHub)" <gi...@apache.org>.
mahdibh commented on issue #8642:
URL: https://github.com/apache/iceberg/issues/8642#issuecomment-1735852931

   I think the lack of consistency is problematic (breaks the principle of least surprise). I'm not super familiar with iceberg internals, but here is one way that this issue could be addressed while still properly dealing with #218 
   
   1. when specifying a delete handler on any of the `Transaction` methods returning an implementation of `SnapshotUpdate`, the `Transaction` implementation will store that on the side and precondition that the instance is unique across the various methods. This enforces a single custom delete handler for all the operation which seems like a reasonable requirement for a `Transaction`. Another option is that the handler can be set on the `Transaction` object which may seem more natural (ie, you can only set it on the transaction but you can't set it on individual operations). 
   2. the current implementation will call the handler on the final `Set<String>` of files that need to be deleted.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] nastra commented on issue #8642: Cannot specify custom delete handler when using a DeleteFiles in a Transaction

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #8642:
URL: https://github.com/apache/iceberg/issues/8642#issuecomment-1735045482

   According to https://github.com/apache/iceberg/blob/master/core/src/test/java/org/apache/iceberg/TestTransaction.java#L658-L668 this seems to be expected behavior. The reasoning is described in https://github.com/apache/iceberg/pull/218


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Cannot specify custom delete handler when using a DeleteFiles in a Transaction [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8642:
URL: https://github.com/apache/iceberg/issues/8642#issuecomment-2362461601

   This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org