You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mahdibh (via GitHub)" <gi...@apache.org> on 2023/09/25 16:15:38 UTC
[GitHub] [iceberg] mahdibh opened a new issue, #8642: Cannot specify custom delete handler when using a DeleteFiles in a Transaction
mahdibh opened a new issue, #8642:
URL: https://github.com/apache/iceberg/issues/8642
### Apache Iceberg version
1.2.0
### Query engine
None
### Please describe the bug 🐞
I can specify my own delete handler when using a DeleteFiles operation directly (through `table.newDelete()`). However, when doing it through a transaction (ie, `table.newTransaction().newDelete()`) it fails with the following exception
```
java.lang.IllegalArgumentException: Cannot set delete callback more than once
at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
at org.apache.iceberg.SnapshotProducer.deleteWith(SnapshotProducer.java:164)
at IcebergDeleteTest.main(IcebergDeleteTest.java:42)
```
I would expect the delete to work the same way within a transaction. If the internal code needs to do cleanups, those should be handled separately from the actual cleanups that happen as a result of the delete operation.
Here is a full repro with the successful case first, then the failed one
```
import org.apache.hadoop.conf.Configuration;
import org.apache.iceberg.DeleteFiles;
import org.apache.iceberg.Schema;
import org.apache.iceberg.Table;
import org.apache.iceberg.Transaction;
import org.apache.iceberg.catalog.Catalog;
import org.apache.iceberg.catalog.TableIdentifier;
import org.apache.iceberg.hadoop.HadoopCatalog;
import org.apache.iceberg.types.Types;
public class IcebergDeleteTest {
public static void main(String[] args) {
// Specify the local directory where the Hadoop catalog will store metadata
String catalogPath = "file:///tmp/catalog";
// Create a Hadoop Catalog
Catalog catalog = new HadoopCatalog(new Configuration(), catalogPath);
// Define the table identifier
TableIdentifier tableIdentifier = TableIdentifier.of("my_database", "my_table");
// Create a schema for your data
Schema schema = new Schema(Types.NestedField.optional(1, "id", Types.IntegerType.get()));
// Create a table
Table table = catalog.createTable(tableIdentifier, schema);
// delete directly (works fine)
DeleteFiles deleteFiles = table.newDelete();
deleteFiles
.deleteWith(
(path) -> {
System.err.println("deleting file " + path);
})
.deleteFile("file://foo1")
.commit();
// delete through a transaction (throws exception)
Transaction transaction = table.newTransaction();
deleteFiles = transaction.newDelete();
deleteFiles
.deleteWith(
(path) -> {
System.err.println("deleting file " + path);
})
.deleteFile("file://foo2")
.commit();
transaction.commitTransaction();
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] mahdibh commented on issue #8642: Cannot specify custom delete handler when using a DeleteFiles in a Transaction
Posted by "mahdibh (via GitHub)" <gi...@apache.org>.
mahdibh commented on issue #8642:
URL: https://github.com/apache/iceberg/issues/8642#issuecomment-1735852931
I think the lack of consistency is problematic (breaks the principle of least surprise). I'm not super familiar with iceberg internals, but here is one way that this issue could be addressed while still properly dealing with #218
1. when specifying a delete handler on any of the `Transaction` methods returning an implementation of `SnapshotUpdate`, the `Transaction` implementation will store that on the side and precondition that the instance is unique across the various methods. This enforces a single custom delete handler for all the operation which seems like a reasonable requirement for a `Transaction`. Another option is that the handler can be set on the `Transaction` object which may seem more natural (ie, you can only set it on the transaction but you can't set it on individual operations).
2. the current implementation will call the handler on the final `Set<String>` of files that need to be deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #8642: Cannot specify custom delete handler when using a DeleteFiles in a Transaction
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #8642:
URL: https://github.com/apache/iceberg/issues/8642#issuecomment-1735045482
According to https://github.com/apache/iceberg/blob/master/core/src/test/java/org/apache/iceberg/TestTransaction.java#L658-L668 this seems to be expected behavior. The reasoning is described in https://github.com/apache/iceberg/pull/218
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Cannot specify custom delete handler when using a DeleteFiles in a Transaction [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8642:
URL: https://github.com/apache/iceberg/issues/8642#issuecomment-2362461601
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org