You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "raunaqmorarka (via GitHub)" <gi...@apache.org> on 2026/08/18 15:59:02 UTC
[PR] Core: Retry REST requests rejected with AWS throttling error type [iceberg]
raunaqmorarka opened a new pull request, #17710:
URL: https://github.com/apache/iceberg/pull/17710
AWS services can signal throttling with a 400 status instead of 429. The AWS Glue Iceberg REST endpoint returns HTTP 400 with header `x-amzn-errortype: ThrottlingException` and body `{"message":"Rate exceeded"}` under load. The client maps that to a non-retryable `BadRequestException`, so concurrent commits fail instead of backing off.
This change makes `ExponentialHttpRequestRetryStrategy` retry a response whose `x-amzn-errortype` header reports a throttling error, using the existing exponential backoff and max-retries bound. A throttled request was rejected before execution, so retrying is safe for non-idempotent requests as well.
Observed against `https://glue.us-east-2.amazonaws.com/iceberg` with concurrent table commits and reproduced with a burst of signed loadTable requests (448 of 1800 throttled, all HTTP 400 with the header above).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry REST requests rejected with AWS throttling error type [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #17710:
URL: https://github.com/apache/iceberg/pull/17710#discussion_r3814419559
##########
core/src/main/java/org/apache/iceberg/rest/ExponentialHttpRequestRetryStrategy.java:
##########
@@ -145,14 +155,42 @@ public boolean retryRequest(HttpResponse response, int execCount, HttpContext co
// A retry is permitted if all the following conditions are met:
// 1. The maximum retry count has not been exceeded.
- // 2. The response code is considered retryable, for one of the following reasons:
- // - It's in a predefined list of retriable codes.
+ // 2. The response is considered retryable, for one of the following reasons:
+ // - The response code is in a predefined list of retriable codes.
// - The request is idempotent, and the response code indicates a retry is safe.
// - The response code is '503 Service Unavailable' and includes a 'Retry-After' header.
+ // - The response reports an AWS throttling error, which can use a 400 response code.
return execCount <= maxRetries
&& (retriableCodes.contains(response.getCode())
|| shouldRetryIdempotent(request, response.getCode())
- || is503Retryable);
+ || is503Retryable
+ || isAwsThrottling(response));
+ }
+
+ private boolean isAwsThrottling(HttpResponse response) {
+ if (response.getCode() < HttpStatus.SC_BAD_REQUEST) {
+ return false;
+ }
+
+ Header header = response.getFirstHeader(AWS_ERROR_TYPE_HEADER);
+ if (header == null || header.getValue() == null) {
+ return false;
+ }
+
+ // the header value may be formatted as 'ErrorType', 'ErrorType:uri', 'namespace#ErrorType',
+ // or 'namespace#ErrorType:uri'
+ String errorType = header.getValue();
+ int colon = errorType.indexOf(':');
+ if (colon >= 0) {
+ errorType = errorType.substring(0, colon);
+ }
+
+ int hash = errorType.lastIndexOf('#');
+ if (hash >= 0) {
+ errorType = errorType.substring(hash + 1);
+ }
+
+ return AWS_THROTTLING_ERROR_TYPES.contains(errorType);
}
Review Comment:
The fix looks correct to me, but at the wrong layer. This code is generic HTTP handling and should not do AWS specific handling. One way to model this is to configure an optional error handler class that can be injected, where you can do AWS specific handling. (similar to `HTTPClient.REST_TLS_CONFIGURER`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry REST requests rejected with AWS throttling error type [iceberg]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] commented on PR #17710:
URL: https://github.com/apache/iceberg/pull/17710#issuecomment-6645932717
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry REST requests rejected with AWS throttling error type [iceberg]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on PR #17710:
URL: https://github.com/apache/iceberg/pull/17710#issuecomment-5419105948
I believe Glue is supposed to return 429 in this case to align with IRC spec, I'll ask someone to investigate further and fix it. is there a GH issue tracking this bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry REST requests rejected with AWS throttling error type [iceberg]
Posted by "raunaqmorarka (via GitHub)" <gi...@apache.org>.
raunaqmorarka commented on code in PR #17710:
URL: https://github.com/apache/iceberg/pull/17710#discussion_r3810229883
##########
core/src/main/java/org/apache/iceberg/rest/ExponentialHttpRequestRetryStrategy.java:
##########
@@ -145,14 +155,42 @@ public boolean retryRequest(HttpResponse response, int execCount, HttpContext co
// A retry is permitted if all the following conditions are met:
// 1. The maximum retry count has not been exceeded.
- // 2. The response code is considered retryable, for one of the following reasons:
- // - It's in a predefined list of retriable codes.
+ // 2. The response is considered retryable, for one of the following reasons:
+ // - The response code is in a predefined list of retriable codes.
// - The request is idempotent, and the response code indicates a retry is safe.
// - The response code is '503 Service Unavailable' and includes a 'Retry-After' header.
+ // - The response reports an AWS throttling error, which can use a 400 response code.
return execCount <= maxRetries
&& (retriableCodes.contains(response.getCode())
|| shouldRetryIdempotent(request, response.getCode())
- || is503Retryable);
+ || is503Retryable
+ || isAwsThrottling(response));
+ }
+
+ private boolean isAwsThrottling(HttpResponse response) {
+ if (response.getCode() < HttpStatus.SC_BAD_REQUEST) {
+ return false;
+ }
+
+ Header header = response.getFirstHeader(AWS_ERROR_TYPE_HEADER);
+ if (header == null || header.getValue() == null) {
+ return false;
+ }
+
+ // the header value may be formatted as 'ErrorType', 'ErrorType:uri', 'namespace#ErrorType',
+ // or 'namespace#ErrorType:uri'
+ String errorType = header.getValue();
+ int colon = errorType.indexOf(':');
+ if (colon >= 0) {
+ errorType = errorType.substring(0, colon);
+ }
+
+ int hash = errorType.lastIndexOf('#');
+ if (hash >= 0) {
+ errorType = errorType.substring(hash + 1);
+ }
+
+ return AWS_THROTTLING_ERROR_TYPES.contains(errorType);
}
Review Comment:
It would indeed be nicer if Glue fixed it, do we have any AWS folks to tag here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry REST requests rejected with AWS throttling error type [iceberg]
Posted by "raunaqmorarka (via GitHub)" <gi...@apache.org>.
raunaqmorarka commented on code in PR #17710:
URL: https://github.com/apache/iceberg/pull/17710#discussion_r3815544730
##########
core/src/main/java/org/apache/iceberg/rest/ExponentialHttpRequestRetryStrategy.java:
##########
@@ -145,14 +155,42 @@ public boolean retryRequest(HttpResponse response, int execCount, HttpContext co
// A retry is permitted if all the following conditions are met:
// 1. The maximum retry count has not been exceeded.
- // 2. The response code is considered retryable, for one of the following reasons:
- // - It's in a predefined list of retriable codes.
+ // 2. The response is considered retryable, for one of the following reasons:
+ // - The response code is in a predefined list of retriable codes.
// - The request is idempotent, and the response code indicates a retry is safe.
// - The response code is '503 Service Unavailable' and includes a 'Retry-After' header.
+ // - The response reports an AWS throttling error, which can use a 400 response code.
return execCount <= maxRetries
&& (retriableCodes.contains(response.getCode())
|| shouldRetryIdempotent(request, response.getCode())
- || is503Retryable);
+ || is503Retryable
+ || isAwsThrottling(response));
+ }
+
+ private boolean isAwsThrottling(HttpResponse response) {
+ if (response.getCode() < HttpStatus.SC_BAD_REQUEST) {
+ return false;
+ }
+
+ Header header = response.getFirstHeader(AWS_ERROR_TYPE_HEADER);
+ if (header == null || header.getValue() == null) {
+ return false;
+ }
+
+ // the header value may be formatted as 'ErrorType', 'ErrorType:uri', 'namespace#ErrorType',
+ // or 'namespace#ErrorType:uri'
+ String errorType = header.getValue();
+ int colon = errorType.indexOf(':');
+ if (colon >= 0) {
+ errorType = errorType.substring(0, colon);
+ }
+
+ int hash = errorType.lastIndexOf('#');
+ if (hash >= 0) {
+ errorType = errorType.substring(hash + 1);
+ }
+
+ return AWS_THROTTLING_ERROR_TYPES.contains(errorType);
}
Review Comment:
Would this shape work: a small interface in core, say `HttpResponseRetryPredicate`, that `ExponentialHttpRequestRetryStrategy` consults in addition to its built-in rules. `HTTPClient` would load the AWS implementation from iceberg-aws reflectively whenever SigV4 auth is enabled, the same way `AuthManagers` resolves the SigV4 `AuthManager` by hardcoded class name. The `x-amzn-ErrorType` parsing moves to that implementation, so Glue and S3 Tables endpoints get the retry automatically and this class stays generic. The check only fires when the response carries the AWS error type header, so other servers are unaffected.
If this direction sounds right I'll restructure the PR accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry REST requests rejected with AWS throttling error type [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17710:
URL: https://github.com/apache/iceberg/pull/17710#issuecomment-6457210747
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry REST requests rejected with AWS throttling error type [iceberg]
Posted by "raunaqmorarka (via GitHub)" <gi...@apache.org>.
raunaqmorarka commented on PR #17710:
URL: https://github.com/apache/iceberg/pull/17710#issuecomment-5842888818
@CTTY any updates on this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry REST requests rejected with AWS throttling error type [iceberg]
Posted by "raunaqmorarka (via GitHub)" <gi...@apache.org>.
raunaqmorarka commented on PR #17710:
URL: https://github.com/apache/iceberg/pull/17710#issuecomment-5425802019
Filed #17824 to track the Glue-side deviation (400 + `x-amzn-ErrorType: ThrottlingException` instead of 429). Reproduced against the Glue IRC endpoint in us-east-2: hammering `loadTable` returns HTTP 400 with body `{"message":"Rate exceeded"}` and the throttling error type only in the header. This PR is the client-side workaround until Glue returns 429.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry REST requests rejected with AWS throttling error type [iceberg]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on PR #17710:
URL: https://github.com/apache/iceberg/pull/17710#issuecomment-5365648500
It's the good old 400 vs 429 for throttling exception problem :) I don't see this behavior of Glue documented anywhere and will need to ask around and circle back later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry REST requests rejected with AWS throttling error type [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #17710:
URL: https://github.com/apache/iceberg/pull/17710#discussion_r3808765178
##########
core/src/main/java/org/apache/iceberg/rest/ExponentialHttpRequestRetryStrategy.java:
##########
@@ -145,14 +155,42 @@ public boolean retryRequest(HttpResponse response, int execCount, HttpContext co
// A retry is permitted if all the following conditions are met:
// 1. The maximum retry count has not been exceeded.
- // 2. The response code is considered retryable, for one of the following reasons:
- // - It's in a predefined list of retriable codes.
+ // 2. The response is considered retryable, for one of the following reasons:
+ // - The response code is in a predefined list of retriable codes.
// - The request is idempotent, and the response code indicates a retry is safe.
// - The response code is '503 Service Unavailable' and includes a 'Retry-After' header.
+ // - The response reports an AWS throttling error, which can use a 400 response code.
return execCount <= maxRetries
&& (retriableCodes.contains(response.getCode())
|| shouldRetryIdempotent(request, response.getCode())
- || is503Retryable);
+ || is503Retryable
+ || isAwsThrottling(response));
+ }
+
+ private boolean isAwsThrottling(HttpResponse response) {
+ if (response.getCode() < HttpStatus.SC_BAD_REQUEST) {
+ return false;
+ }
+
+ Header header = response.getFirstHeader(AWS_ERROR_TYPE_HEADER);
+ if (header == null || header.getValue() == null) {
+ return false;
+ }
+
+ // the header value may be formatted as 'ErrorType', 'ErrorType:uri', 'namespace#ErrorType',
+ // or 'namespace#ErrorType:uri'
+ String errorType = header.getValue();
+ int colon = errorType.indexOf(':');
+ if (colon >= 0) {
+ errorType = errorType.substring(0, colon);
+ }
+
+ int hash = errorType.lastIndexOf('#');
+ if (hash >= 0) {
+ errorType = errorType.substring(hash + 1);
+ }
+
+ return AWS_THROTTLING_ERROR_TYPES.contains(errorType);
}
Review Comment:
I wonder if this is something glue should fix instead ? 400 means malformed request or something wrong in the request https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L5506
i understand spec doesn't clearly state that 429 is the error code to throw, but this seems incorrect to me to rely on response header that too of 400 status
i would recommend keeping this class as generic to rest as possible and not add implementation specific semantics here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry REST requests rejected with AWS throttling error type [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17710:
URL: https://github.com/apache/iceberg/pull/17710#issuecomment-5841492661
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry REST requests rejected with AWS throttling error type [iceberg]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] closed pull request #17710: Core: Retry REST requests rejected with AWS throttling error type
URL: https://github.com/apache/iceberg/pull/17710
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org