You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "astarrr (via GitHub)" <gi...@apache.org> on 2025/07/18 13:59:51 UTC
[PR] AWS: Make the number of retries for metadata location refresh configurable [iceberg]
astarrr opened a new pull request, #13594:
URL: https://github.com/apache/iceberg/pull/13594
Closes: #13529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Make the number of retries for metadata location refresh configurable [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13594:
URL: https://github.com/apache/iceberg/pull/13594#issuecomment-3315386942
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Make the number of retries for metadata location refresh configurable [iceberg]
Posted by "astarrr (via GitHub)" <gi...@apache.org>.
astarrr commented on PR #13594:
URL: https://github.com/apache/iceberg/pull/13594#issuecomment-3209338830
trying to write a custom AWS client to see the effort compared to setting a conf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Make the number of retries for metadata location refresh configurable [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #13594:
URL: https://github.com/apache/iceberg/pull/13594#discussion_r2222583250
##########
docs/docs/configuration.md:
##########
@@ -178,6 +178,7 @@ The HMS table locking is a 2-step process:
| iceberg.hive.metadata-refresh-max-retries | 2 | Maximum number of retries when the metadata file is missing |
| iceberg.hive.table-level-lock-evict-ms | 600000 (10 min) | The timeout for the JVM table lock is |
| iceberg.engine.hive.lock-enabled | true | Use HMS locks to ensure atomicity of commits |
+| iceberg.glue.metadata-refresh-max-retries | 20 | Maximum number of retries when the metadata file is missing |
Review Comment:
I also don't have the full history here, but 20 seems like a lot (but I do see that's what the default has been for quite some time)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Make the number of retries for metadata location refresh configurable [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13594:
URL: https://github.com/apache/iceberg/pull/13594#issuecomment-3342136658
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Make the number of retries for metadata location refresh configurable [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #13594: AWS: Make the number of retries for metadata location refresh configurable
URL: https://github.com/apache/iceberg/pull/13594
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Make the number of retries for metadata location refresh configurable [iceberg]
Posted by "astarrr (via GitHub)" <gi...@apache.org>.
astarrr commented on code in PR #13594:
URL: https://github.com/apache/iceberg/pull/13594#discussion_r2222631229
##########
docs/docs/configuration.md:
##########
@@ -178,6 +178,7 @@ The HMS table locking is a 2-step process:
| iceberg.hive.metadata-refresh-max-retries | 2 | Maximum number of retries when the metadata file is missing |
| iceberg.hive.table-level-lock-evict-ms | 600000 (10 min) | The timeout for the JVM table lock is |
| iceberg.engine.hive.lock-enabled | true | Use HMS locks to ensure atomicity of commits |
+| iceberg.glue.metadata-refresh-max-retries | 20 | Maximum number of retries when the metadata file is missing |
Review Comment:
Correct, the retries addresses other problems. From my experience one has been that the credentials were expired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] AWS: Make the number of retries for metadata location refresh configurable [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #13594:
URL: https://github.com/apache/iceberg/pull/13594#discussion_r2222566699
##########
docs/docs/configuration.md:
##########
@@ -178,6 +178,7 @@ The HMS table locking is a 2-step process:
| iceberg.hive.metadata-refresh-max-retries | 2 | Maximum number of retries when the metadata file is missing |
| iceberg.hive.table-level-lock-evict-ms | 600000 (10 min) | The timeout for the JVM table lock is |
| iceberg.engine.hive.lock-enabled | true | Use HMS locks to ensure atomicity of commits |
+| iceberg.glue.metadata-refresh-max-retries | 20 | Maximum number of retries when the metadata file is missing |
Review Comment:
Btw just to make sure we're on the same page about addressing the right problem, the code path that's being changed here will actually _stop_ retrying when the metadata file is missing; the retries address any other kind of failure.
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java:
##########
@@ -77,6 +78,10 @@ class GlueTableOperations extends BaseMetastoreTableOperations {
private final Object hadoopConf;
private final LockManager lockManager;
private FileIO fileIO;
+ private final int metadataRefreshMaxRetries;
+ private static final String GLUE_ICEBERG_METADATA_REFRESH_MAX_RETRIES =
+ "iceberg.glue.metadata-refresh-max-retries";
+ private static final int GLUE_ICEBERG_METADATA_REFRESH_MAX_RETRIES_DEFAULT = 20;
Review Comment:
If we were to do this, I'd say this should probably be something like a Glue catalog property defined in `AwsProperties` instead of through Hadoop conf.
But I still think we should instead try to avoid another configuration here and just use the ability to plug in your own Glue client with your desired behavior as the way to address this.
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java:
##########
@@ -172,6 +172,10 @@ void initialize(
LockManager lock,
Map<String, String> catalogProps) {
this.catalogProperties = catalogProps;
+ if (hadoopConf == null) {
+ LOG.warn("No Hadoop Configuration was set, using the default environment Configuration");
+ this.hadoopConf = new Configuration();
+ }
Review Comment:
See my comment below about how if we were to do this, I don't think it should be through hadoop conf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org