You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "wuwenchi (via GitHub)" <gi...@apache.org> on 2025/04/18 05:51:59 UTC
[I] RewriteDataFiles with merging equality deletes [iceberg]
wuwenchi opened a new issue, #12838:
URL: https://github.com/apache/iceberg/issues/12838
### Apache Iceberg version
1.5.0
### Query engine
Spark
### Please describe the bug 🐞
We use Flink to write data into an Iceberg table with a primary key, and then use Spark to rewrite the Iceberg table. Although the data files will be merged into a large file as expected, the corresponding delete files are never deleted.
The main reason lies here:
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L914-L925
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/ManifestFilterManager.java#L427-L430
For delete files, only those smaller than the `minDataSequenceNumber` will be deleted.
The `minDataSequenceNumber` is the sequence number of the smallest data file in the current table, and no judgment is made for different partitions.
So the sequence number of partition 'a' may affect the deletion of delete files in partition 'b'.
When performing merges on all newly inserted partitions (such as "c","d","e","f") in the future, the delete files will no longer be deleted.
Is this a problem? Or how can I work around this issue? It's not feasible to perform a rewrite on the entire table because there is too much data. Can I use the `remove-dangling-deletes` option supported in version 1.8 to delete them? However, we are currently using version 1.5, and it's not easy to upgrade our environment. Are there any other good solutions?
Here are the steps to reproduce the issue:
with flink:
```
CREATE TABLE `xx10` (
id BIGINT COMMENT 'unique id',
data INT NOT NULL,
par STRING NOT NULL,
PRIMARY KEY(`id`, `par`) NOT ENFORCED
) PARTITIONED BY(`par`) WITH ('format-version'='2', 'write.upsert.enabled'='true');
insert into xx10 values (1,1,'a'), (2,1,'a'), (3,1,'a');
insert into xx10 values (1,1,'b'), (2,1,'b'), (3,1,'b');
```
with spark:
```
CALL catalog.system.rewrite_data_files(table => 'db.xx10', where => 'par ="a"', options => map('rewrite-all', 'true'));
```
the delete files in partition 'a' will be deleted.
with flink:
```
insert into xx10 values (1,1,'c'), (2,1,'c'), (3,1,'c');
```
with spark:
```
CALL catalog.system.rewrite_data_files(table => 'db.xx10', where => 'par ="b"', options => map('rewrite-all', 'true'));
```
the delete files in partition 'b' will always exist.
At this point, when I'm performing a merge operation on partition 'c', the delete files in partition 'c' won't be deleted either.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] RewriteDataFiles with merging equality deletes [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12838:
URL: https://github.com/apache/iceberg/issues/12838#issuecomment-3514446092
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] RewriteDataFiles with merging equality deletes [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12838: RewriteDataFiles with merging equality deletes
URL: https://github.com/apache/iceberg/issues/12838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] RewriteDataFiles with merging equality deletes [iceberg]
Posted by "shanielh (via GitHub)" <gi...@apache.org>.
shanielh commented on issue #12838:
URL: https://github.com/apache/iceberg/issues/12838#issuecomment-2857098062
It's a known issue and, unfortunately, by design. Many underlying design decisions fail to take streaming data into account.
We solved it in Upsolver by reading the manifests, removing the delete files that are not needed anymore (using the minSequenceNumber of each partition).
I hope your issue will help to solve it in Iceberg itself (although I'm pretty sure another one was closed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] RewriteDataFiles with merging equality deletes [iceberg]
Posted by "shanielh (via GitHub)" <gi...@apache.org>.
shanielh commented on issue #12838:
URL: https://github.com/apache/iceberg/issues/12838#issuecomment-2873680943
> Amoro tried to resolve this issue by [another may](https://github.com/apache/amoro/blob/master/amoro-ams/src/main/java/org/apache/amoro/server/utils/IcebergTableUtil.java#L148)
>
> It determines which delete files are dangling by comparing the delete files involved in the table scan with the delete files returned in the table entries table.
>
> It will incur higher resource consumption, but it can precisely identify all dangling delete files.
Using scan is slow since it returns the same delete files over and over due to the nature of Iceberg. You should probably "run scan" by going over the manifest files and indexing by partition the minimum data sequence number and then delete per partition delete files that their sequence number is lower than that sequence number, it's a bit more tricky since partitioning in iceberg can evolve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] RewriteDataFiles with merging equality deletes [iceberg]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on issue #12838:
URL: https://github.com/apache/iceberg/issues/12838#issuecomment-2857488110
Amoro tried to resolve this issue by [another may](https://github.com/apache/amoro/blob/master/amoro-ams/src/main/java/org/apache/amoro/server/utils/IcebergTableUtil.java#L148)
It determines which delete files are dangling by comparing the delete files involved in the table scan with the delete files returned in the table entries table.
It will incur higher resource consumption, but it can precisely identify all dangling delete files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] RewriteDataFiles with merging equality deletes [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12838:
URL: https://github.com/apache/iceberg/issues/12838#issuecomment-3578168152
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org