You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jotarada (via GitHub)" <gi...@apache.org> on 2025/03/20 12:40:37 UTC
[I] rewrite_table_path looking for delete files that do not exists [iceberg]
jotarada opened a new issue, #12586:
URL: https://github.com/apache/iceberg/issues/12586
### Apache Iceberg version
1.8.1 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
To overcome the issue reported at https://github.com/apache/iceberg/issues/12554 I tried to compact data files and then expire all snapshots, so we should only have data files and metadata files to move and no delete files.
But when I run rewrite_table_path after those steps i got `gs://bucket/dir/data/id_trunc_2=6f/20250319_111721_22254_g4jwb-90e36525-e698-4d52-82a9-9d6700b951f3.parquet `
This was a delete file that is not anymore part of the table, and is not even present anymore, but for some reason the rewrite_table_path still tries to rewrite it.
Steps to reproduce
* Have some table we some data
* Delete a record from the table using MOR (trino does it by default)
* Compact data files **twice**
```
CALL iceberg.system.rewrite_data_files(
table =>'schema.table',
options => map('remove-dangling-deletes', 'true', 'delete-file-threshold', '0')
)
```
* Expire snapshots
```
CALL iceberg.system.expire_snapshots(
table => 'schema.table',
older_than => TIMESTAMP ' NOW
)
```
* Try to table re-write
```
CALL iceberg.system.rewrite_table_path(
table => 'schema.table',
source_prefix => 'current_location',
target_prefix => 'new_location'
)
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewrite_table_path looking for delete files that do not exists [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12586:
URL: https://github.com/apache/iceberg/issues/12586#issuecomment-3331142992
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewrite_table_path looking for delete files that do not exists [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12586:
URL: https://github.com/apache/iceberg/issues/12586#issuecomment-3387847427
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewrite_table_path looking for delete files that do not exists [iceberg]
Posted by "wobu (via GitHub)" <gi...@apache.org>.
wobu commented on issue #12586:
URL: https://github.com/apache/iceberg/issues/12586#issuecomment-3817160443
i think the issue lies here: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java#L468
it thould not add a DeleteFile to the Rewrite plan if it isn't live anymore. Chances are high that it got cleaned up already.
Nevertheless, if the file isn't added to the CopyPlan, there is also no need for rewriting it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewrite_table_path looking for delete files that do not exists [iceberg]
Posted by "jotarada (via GitHub)" <gi...@apache.org>.
jotarada commented on issue #12586:
URL: https://github.com/apache/iceberg/issues/12586#issuecomment-2762138250
I used spark 3.5.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewrite_table_path looking for delete files that do not exists [iceberg]
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on issue #12586:
URL: https://github.com/apache/iceberg/issues/12586#issuecomment-3825173775
> i think the issue lies here: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java#L468
>
> it thould not add a DeleteFile to the Rewrite plan if it isn't live anymore. Chances are high that it got cleaned up already.
>
> Nevertheless, if the file isn't added to the CopyPlan, there is also no need for rewriting it.
>
> [@dramaticlly](https://github.com/dramaticlly) what's your opinion?
Thanks for the ping! I recall discussing this with @szehon-ho and @flyrain previously.
When rewriting file paths in delete manifests, we deliberately maintain the original entry count to preserve the manifest's semantic meaning to. Here's how it works:
- Live position delete entries: We copy the actual position delete files (with path rewritten) to the new location and update their corresponding entries in the new delete manifests.
- Non-live (deleted) position delete entries: We do not copy these files (since they're no longer needed), but we still rewrite their entries in the new delete manifests to preserve the manifest structure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewrite_table_path looking for delete files that do not exists [iceberg]
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on issue #12586:
URL: https://github.com/apache/iceberg/issues/12586#issuecomment-2762001759
@jotarada which spark version do you use for the rewrite table path? I think the problem looks like https://github.com/apache/iceberg/pull/12006 and is released with iceberg 1.8.1 in spark 3.5, but recently it gets backported to spark 3.4 in https://github.com/apache/iceberg/pull/12568, which is not released yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewrite_table_path looking for delete files that do not exists [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12586: rewrite_table_path looking for delete files that do not exists
URL: https://github.com/apache/iceberg/issues/12586
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org