You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "gaborgsomogyi (via GitHub)" <gi...@apache.org> on 2026/05/22 15:41:33 UTC
[PR] [FLINK-39737][s3] Switch to StandardRetryStrategy with configurable backoff for throttle-aware retry [flink]
gaborgsomogyi opened a new pull request, #28236:
URL: https://github.com/apache/flink/pull/28236
## What is the purpose of the change
The `flink-s3-fs-native` connector was using the deprecated `RetryPolicy` API from AWS SDK v2 in legacy retry mode. The legacy mode does not distinguish throttling errors (HTTP 429) from transient server errors, applies no token-bucket circuit breaking, and has been superseded by the `RetryStrategy` API since SDK v2.25. S3-compatible storage systems commonly signal throttling via HTTP 429, which the legacy mode silently dropped without retrying.
This pull request replaces the deprecated `RetryPolicy` with `StandardRetryStrategy` from the non-deprecated AWS SDK v2 retries API. The standard strategy natively handles both HTTP 429 and 503 as throttling events with a dedicated backoff path, applies full-jitter exponential backoff separately for throttle and non-throttle errors, and uses a token-bucket circuit breaker to prevent retry storms under sustained load. The key backoff parameters are exposed as Flink configuration options so operators can tune retry behavior for their storage backend.
## Brief change log
- Replace deprecated `RetryPolicy.builder().numRetries(n)` with `StandardRetryStrategy` via `ClientOverrideConfiguration.retryStrategy()` in `S3ClientProvider`
- Add three new Flink config options (defaults match the SDK standard strategy defaults):
- `s3.retry.base-delay` (default: 100ms) — backoff base delay for non-throttle retries
- `s3.retry.throttle.base-delay` (default: 1s) — backoff base delay for throttle retries (HTTP 429/503)
- `s3.retry.max-backoff` (default: 20s) — shared exponential backoff cap for both retry paths
- Builder defaults for the new options reference the `ConfigOption` default values directly, keeping a single source of truth
- Existing `s3.retry.max-num-retries` (default: 3) is preserved and maps to `maxAttempts = maxRetries + 1`
## Verifying this change
This change added tests and can be verified as follows:
- Added 6 tests to `NativeS3FileSystemFactoryTest` covering default values and explicit configuration for each of the three new `Duration` config options, following the same pattern as the existing `testMaxRetriesDefault` / `testMaxRetriesExplicitlyConfigured` tests
- Added 1 test to `S3ClientProviderTest` (`testRetryBuilderDefaultsMatchConfigOptions`) verifying that `Builder` field defaults and `ConfigOption` defaults stay in sync
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no (`software.amazon.awssdk:retries` was already a transitive dependency via `sdk-core`)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: yes
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? JavaDocs on the new `ConfigOption` fields in `NativeS3FileSystemFactory`
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (Claude Sonnet 4.6)
<!--
Generated-by: Claude Sonnet 4.6
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39737][s3] Switch to StandardRetryStrategy with configurable backoff for throttle-aware retry [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #28236:
URL: https://github.com/apache/flink/pull/28236#discussion_r3296119247
##########
flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/S3ClientProvider.java:
##########
@@ -141,6 +148,13 @@ private S3ClientProvider(
this.checksumValidation = checksumValidation;
this.maxConnections = maxConnections;
this.maxRetries = maxRetries;
+ this.retryBaseDelay =
+ Preconditions.checkNotNull(retryBaseDelay, "retryBaseDelay must not be null");
+ this.retryThrottleBaseDelay =
+ Preconditions.checkNotNull(
+ retryThrottleBaseDelay, "retryThrottleBaseDelay must not be null");
+ this.retryMaxBackoff =
+ Preconditions.checkNotNull(retryMaxBackoff, "retryMaxBackoff must not be null");
Review Comment:
Invalid values are possible. Add condition check for `negative durations`, `zero durations`, `maxBackoff < baseDelay`, `maxBackoff < throttleBaseDelay`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39737][s3] Switch to StandardRetryStrategy with configurable backoff for throttle-aware retry [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28236:
URL: https://github.com/apache/flink/pull/28236#issuecomment-4520182589
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f617c737d4541f1cd7491c35f3dfdcd5ea89eb3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2f617c737d4541f1cd7491c35f3dfdcd5ea89eb3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f617c737d4541f1cd7491c35f3dfdcd5ea89eb3 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39737][s3] Switch to StandardRetryStrategy with configurable backoff for throttle-aware retry [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on PR #28236:
URL: https://github.com/apache/flink/pull/28236#issuecomment-4543578507
\`StandardRetryStrategy\` is the SDK's recommended general-purpose strategy, which is the primary reason for this choice. Beyond that, the SDK's own documentation explicitly cautions against \`AdaptiveRetryStrategy\` when a single client serves multiple resources: *"throttling or outages associated with one resource will result in increased latency and failures when accessing all other resources via the same client."* Flink's S3 client is shared across concurrent operations (checkpoint writes, reads, metadata calls), so adaptive mode's rate limiter would cause throttling on one operation to bleed into unrelated ones. \`StandardRetryStrategy\` is the right fit here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39737][s3] Switch to StandardRetryStrategy with configurable backoff for throttle-aware retry [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #28236:
URL: https://github.com/apache/flink/pull/28236#discussion_r3303230704
##########
flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/S3ClientProvider.java:
##########
@@ -141,6 +148,13 @@ private S3ClientProvider(
this.checksumValidation = checksumValidation;
this.maxConnections = maxConnections;
this.maxRetries = maxRetries;
+ this.retryBaseDelay =
+ Preconditions.checkNotNull(retryBaseDelay, "retryBaseDelay must not be null");
+ this.retryThrottleBaseDelay =
+ Preconditions.checkNotNull(
+ retryThrottleBaseDelay, "retryThrottleBaseDelay must not be null");
+ this.retryMaxBackoff =
Review Comment:
Covered.
##########
flink-filesystems/flink-s3-fs-native/src/test/java/org/apache/flink/fs/s3native/NativeS3FileSystemFactoryTest.java:
##########
@@ -178,6 +178,50 @@ void testMaxRetriesExplicitlyConfigured() throws Exception {
assertThat(createFs(config).getClientProvider().getMaxRetries()).isEqualTo(5);
}
+ // --- Retry backoff ---
+
+ @Test
+ void testRetryBaseDelayDefault() throws Exception {
Review Comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39737][s3] Switch to StandardRetryStrategy with configurable backoff for throttle-aware retry [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #28236:
URL: https://github.com/apache/flink/pull/28236#discussion_r3296119247
##########
flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/S3ClientProvider.java:
##########
@@ -141,6 +148,13 @@ private S3ClientProvider(
this.checksumValidation = checksumValidation;
this.maxConnections = maxConnections;
this.maxRetries = maxRetries;
+ this.retryBaseDelay =
+ Preconditions.checkNotNull(retryBaseDelay, "retryBaseDelay must not be null");
+ this.retryThrottleBaseDelay =
+ Preconditions.checkNotNull(
+ retryThrottleBaseDelay, "retryThrottleBaseDelay must not be null");
+ this.retryMaxBackoff =
+ Preconditions.checkNotNull(retryMaxBackoff, "retryMaxBackoff must not be null");
Review Comment:
Invalid values are possible. Add condition check for `negative durations`, `zero durations`, `maxBackoff < baseDelay`, `maxBackoff < throttleBaseDelay`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39737][s3] Switch to StandardRetryStrategy with configurable backoff for throttle-aware retry [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi merged PR #28236:
URL: https://github.com/apache/flink/pull/28236
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39737][s3] Switch to StandardRetryStrategy with configurable backoff for throttle-aware retry [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #28236:
URL: https://github.com/apache/flink/pull/28236#discussion_r3303231938
##########
flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3FileSystemFactory.java:
##########
@@ -243,10 +243,35 @@ public class NativeS3FileSystemFactory implements FileSystemFactory {
.intType()
.defaultValue(3)
.withDescription(
- "Maximum number of retry attempts for failed S3 requests. "
- + "Uses the AWS SDK's default retry strategy (exponential backoff with jitter). "
+ "Maximum number of retries for failed S3 requests (excluding the initial attempt). "
+ "Set to 0 to disable retries.");
+ public static final ConfigOption<Duration> RETRY_BASE_DELAY =
+ ConfigOptions.key("s3.retry.base-delay")
+ .durationType()
+ .defaultValue(Duration.ofMillis(100))
+ .withDescription(
+ "Base delay for exponential backoff on non-throttle retries. "
+ + "Each retry waits a random duration between 0 and "
+ + "min(base-delay * 2^attempt, max-backoff).");
+
+ public static final ConfigOption<Duration> RETRY_THROTTLE_BASE_DELAY =
+ ConfigOptions.key("s3.retry.throttle.base-delay")
+ .durationType()
+ .defaultValue(Duration.ofSeconds(1))
+ .withDescription(
+ "Base delay for exponential backoff on throttle retries (HTTP 429, 503). "
+ + "Each retry waits a random duration between 0 and "
+ + "min(throttle-base-delay * 2^attempt, max-backoff).");
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39737][s3] Switch to StandardRetryStrategy with configurable backoff for throttle-aware retry [flink]
Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #28236:
URL: https://github.com/apache/flink/pull/28236#discussion_r3296120245
##########
flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/S3ClientProvider.java:
##########
@@ -141,6 +148,13 @@ private S3ClientProvider(
this.checksumValidation = checksumValidation;
this.maxConnections = maxConnections;
this.maxRetries = maxRetries;
+ this.retryBaseDelay =
+ Preconditions.checkNotNull(retryBaseDelay, "retryBaseDelay must not be null");
+ this.retryThrottleBaseDelay =
+ Preconditions.checkNotNull(
+ retryThrottleBaseDelay, "retryThrottleBaseDelay must not be null");
+ this.retryMaxBackoff =
Review Comment:
Invalid values are possible. Add condition check for `negative durations`, `zero durations`, `maxBackoff < baseDelay`, `maxBackoff < throttleBaseDelay`
##########
flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3FileSystemFactory.java:
##########
@@ -243,10 +243,35 @@ public class NativeS3FileSystemFactory implements FileSystemFactory {
.intType()
.defaultValue(3)
.withDescription(
- "Maximum number of retry attempts for failed S3 requests. "
- + "Uses the AWS SDK's default retry strategy (exponential backoff with jitter). "
+ "Maximum number of retries for failed S3 requests (excluding the initial attempt). "
+ "Set to 0 to disable retries.");
+ public static final ConfigOption<Duration> RETRY_BASE_DELAY =
+ ConfigOptions.key("s3.retry.base-delay")
+ .durationType()
+ .defaultValue(Duration.ofMillis(100))
+ .withDescription(
+ "Base delay for exponential backoff on non-throttle retries. "
+ + "Each retry waits a random duration between 0 and "
+ + "min(base-delay * 2^attempt, max-backoff).");
+
+ public static final ConfigOption<Duration> RETRY_THROTTLE_BASE_DELAY =
+ ConfigOptions.key("s3.retry.throttle.base-delay")
+ .durationType()
+ .defaultValue(Duration.ofSeconds(1))
+ .withDescription(
+ "Base delay for exponential backoff on throttle retries (HTTP 429, 503). "
+ + "Each retry waits a random duration between 0 and "
+ + "min(throttle-base-delay * 2^attempt, max-backoff).");
Review Comment:
Use `Uses exponential backoff with jitter capped by max-backoff.` rather than a mathematical expression.
In the future, when the logic evolves may lead to misleading.
##########
flink-filesystems/flink-s3-fs-native/src/test/java/org/apache/flink/fs/s3native/NativeS3FileSystemFactoryTest.java:
##########
@@ -178,6 +178,50 @@ void testMaxRetriesExplicitlyConfigured() throws Exception {
assertThat(createFs(config).getClientProvider().getMaxRetries()).isEqualTo(5);
}
+ // --- Retry backoff ---
+
+ @Test
+ void testRetryBaseDelayDefault() throws Exception {
Review Comment:
No tests for actual retry strategy construction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org