You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mirageyjd (via GitHub)" <gi...@apache.org> on 2023/10/27 08:53:29 UTC
[I] Slow RewriteManifests due to Validation of Manifest Entries [iceberg]
mirageyjd opened a new issue, #8932:
URL: https://github.com/apache/iceberg/issues/8932
### Apache Iceberg version
0.13.1
### Query engine
Spark
### Please describe the bug 🐞
We ran `BaseRewriteManifestsSparkAction` action on a large table with 7k+ manifests in Spark, and it took more than an hour unexpectedly. The most time-consuming procedure is to validate that each manifest entry in added manifests has a snapshot id, which is not executed in a distributed manner. Without the validation, the entire action takes less than 2 minutes.
I wonder whether it is necessary to validate snapshot id of each manifest entry in manifests written by `BaseRewriteManifestsSparkAction`. It would be better such validation is optional and can be skipped in`BaseRewriteManifestsSparkAction`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Slow RewriteManifests due to Validation of Manifest Entries [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #8932: Slow RewriteManifests due to Validation of Manifest Entries
URL: https://github.com/apache/iceberg/issues/8932
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Slow RewriteManifests due to Validation of Manifest Entries [iceberg]
Posted by "mirageyjd (via GitHub)" <gi...@apache.org>.
mirageyjd commented on issue #8932:
URL: https://github.com/apache/iceberg/issues/8932#issuecomment-1784723175
Hi @RussellSpitzer. I ran an experiment on a table with 100K data files and collected the total cost of invoking [addManifest](https://github.com/apache/iceberg/blob/b56ec5ed421cb82d0c87c4fdd2a579db40f0e036/core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java#L141).
With `compatibility.snapshot-id-inheritance.enabled=true`, the cost is 2ms.
With `compatibility.snapshot-id-inheritance.enabled=false`, the total cost is 13315ms.
When snapshot id inheritance is disabled, all added manifests need to be validated, and then copied to a new file without changing the content. Copying manifests is expensive since it is executed in a single thread.
https://github.com/apache/iceberg/blob/b56ec5ed421cb82d0c87c4fdd2a579db40f0e036/core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java#L151-L157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Slow RewriteManifests due to Validation of Manifest Entries [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #8932:
URL: https://github.com/apache/iceberg/issues/8932#issuecomment-1782759994
Do you have a flame graph or some evidence of this? My gut would say that would be a trivial check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Slow RewriteManifests due to Validation of Manifest Entries [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8932:
URL: https://github.com/apache/iceberg/issues/8932#issuecomment-2408258032
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Slow RewriteManifests due to Validation of Manifest Entries [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #8932:
URL: https://github.com/apache/iceberg/issues/8932#issuecomment-1784953973
Ah yes, we have numerous threads on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Slow RewriteManifests due to Validation of Manifest Entries [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8932:
URL: https://github.com/apache/iceberg/issues/8932#issuecomment-2378158489
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org