You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "suez1224 (via GitHub)" <gi...@apache.org> on 2025/02/26 07:52:50 UTC
[PR] [FLINK-37366] Allow configurable retry for Kafka topic metadata fetch [flink-connector-kafka]
suez1224 opened a new pull request, #155:
URL: https://github.com/apache/flink-connector-kafka/pull/155
TBD
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37366] Allow configurable retry for Kafka topic metadata fetch [flink-connector-kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #155:
URL: https://github.com/apache/flink-connector-kafka/pull/155#issuecomment-3545502947
This PR is being marked as stale since it has not had any activity in the last 90 days.
If you would like to keep this PR alive, please leave a comment asking for a review.
If the PR has merge conflicts, update it with the latest from the base branch.
If you are having difficulty finding a reviewer, please reach out to the
community, contact details can be found here: https://flink.apache.org/what-is-flink/community/
If this PR is no longer valid or desired, please feel free to close it.
If no activity occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37366] Allow configurable retry for Kafka topic metadata fetch [flink-connector-kafka]
Posted by "AHeise (via GitHub)" <gi...@apache.org>.
AHeise commented on code in PR #155:
URL: https://github.com/apache/flink-connector-kafka/pull/155#discussion_r1993033623
##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/enumerator/subscriber/KafkaSubscriberUtils.java:
##########
@@ -57,12 +65,42 @@ static Map<String, TopicDescription> getTopicMetadata(
}
static Map<String, TopicDescription> getTopicMetadata(
- AdminClient adminClient, Set<String> topicNames) {
- try {
- return adminClient.describeTopics(topicNames).allTopicNames().get();
- } catch (Exception e) {
- throw new RuntimeException(
- String.format("Failed to get metadata for topics %s.", topicNames), e);
+ AdminClient adminClient, Set<String> topicNames, Properties properties) {
+ int maxRetries =
+ KafkaSourceOptions.getOption(
+ properties,
+ KafkaSourceOptions.TOPIC_METADATA_REQUEST_MAX_RETRY,
+ Integer::parseInt);
+ long retryDelay =
+ KafkaSourceOptions.getOption(
+ properties,
+ KafkaSourceOptions.TOPIC_METADATA_REQUEST_RETRY_INTERVAL_MS,
+ Long::parseLong);
+ for (int attempt = 0; attempt <= maxRetries; attempt++) {
+ try {
+ return adminClient.describeTopics(topicNames).allTopicNames().get();
+ } catch (Exception e) {
+ if (attempt == maxRetries) {
+ throw new RuntimeException(
+ String.format("Failed to get metadata for topics %s.", topicNames), e);
+ } else {
+ LOG.warn(
+ "Attempt {} to get metadata for topics {} failed. Retrying in {} ms...",
+ attempt,
+ topicNames,
+ retryDelay);
+ try {
+ TimeUnit.MILLISECONDS.sleep(retryDelay);
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt(); // Restore interrupted state
+ LOG.error("Thread was interrupted during metadata fetch retry delay.", ie);
+ }
+ }
+ }
Review Comment:
Could you please share the failure? If it's something that the AdminClient doesn't deem to be retriable than it's obvious that we need your solution.
It actually may be worth to draft a test that fails without your fix and succeeds with it. We do have some tests that remove the kafka broker (search for `stopBroker`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37366] Allow configurable retry for Kafka topic metadata fetch [flink-connector-kafka]
Posted by "AHeise (via GitHub)" <gi...@apache.org>.
AHeise commented on code in PR #155:
URL: https://github.com/apache/flink-connector-kafka/pull/155#discussion_r1971207127
##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/enumerator/subscriber/KafkaSubscriberUtils.java:
##########
@@ -57,12 +65,42 @@ static Map<String, TopicDescription> getTopicMetadata(
}
static Map<String, TopicDescription> getTopicMetadata(
- AdminClient adminClient, Set<String> topicNames) {
- try {
- return adminClient.describeTopics(topicNames).allTopicNames().get();
- } catch (Exception e) {
- throw new RuntimeException(
- String.format("Failed to get metadata for topics %s.", topicNames), e);
+ AdminClient adminClient, Set<String> topicNames, Properties properties) {
+ int maxRetries =
+ KafkaSourceOptions.getOption(
+ properties,
+ KafkaSourceOptions.TOPIC_METADATA_REQUEST_MAX_RETRY,
+ Integer::parseInt);
+ long retryDelay =
+ KafkaSourceOptions.getOption(
+ properties,
+ KafkaSourceOptions.TOPIC_METADATA_REQUEST_RETRY_INTERVAL_MS,
+ Long::parseLong);
+ for (int attempt = 0; attempt <= maxRetries; attempt++) {
+ try {
+ return adminClient.describeTopics(topicNames).allTopicNames().get();
+ } catch (Exception e) {
+ if (attempt == maxRetries) {
+ throw new RuntimeException(
+ String.format("Failed to get metadata for topics %s.", topicNames), e);
+ } else {
+ LOG.warn(
+ "Attempt {} to get metadata for topics {} failed. Retrying in {} ms...",
+ attempt,
+ topicNames,
+ retryDelay);
+ try {
+ TimeUnit.MILLISECONDS.sleep(retryDelay);
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt(); // Restore interrupted state
+ LOG.error("Thread was interrupted during metadata fetch retry delay.", ie);
+ }
+ }
+ }
Review Comment:
Instead of introducing our own retry logic, can't we reuse what's already implemented in the KafkaAdmin?
```
public static final String RETRIES_CONFIG = "retries";
public static final String RETRIES_DOC = "Setting a value greater than zero will cause the client to resend any request that fails with a potentially transient error." +
" It is recommended to set the value to either zero or `MAX_VALUE` and use corresponding timeout parameters to control how long a client should retry a request.";
public static final String RETRY_BACKOFF_MS_CONFIG = "retry.backoff.ms";
public static final String RETRY_BACKOFF_MS_DOC = "The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.";
```
You should be able to set it with even without your PR like this
```
properties.retries = '10',
properties.retry.backoff.ms = '30000',
```
But that also influences the consumer retry behavior.
We could think about supporting `properties.admin.retry = 10`. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37366] Allow configurable retry for Kafka topic metadata fetch [flink-connector-kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #155:
URL: https://github.com/apache/flink-connector-kafka/pull/155#issuecomment-3668601912
This PR has been closed since it has not had any activity in 120 days.
If you feel like this was a mistake, or you would like to continue working on it,
please feel free to re-open the PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37366] Allow configurable retry for Kafka topic metadata fetch [flink-connector-kafka]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #155:
URL: https://github.com/apache/flink-connector-kafka/pull/155#issuecomment-2684198018
Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37366] Allow configurable retry for Kafka topic metadata fetch [flink-connector-kafka]
Posted by "suez1224 (via GitHub)" <gi...@apache.org>.
suez1224 commented on code in PR #155:
URL: https://github.com/apache/flink-connector-kafka/pull/155#discussion_r1986246516
##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/enumerator/subscriber/KafkaSubscriberUtils.java:
##########
@@ -57,12 +65,42 @@ static Map<String, TopicDescription> getTopicMetadata(
}
static Map<String, TopicDescription> getTopicMetadata(
- AdminClient adminClient, Set<String> topicNames) {
- try {
- return adminClient.describeTopics(topicNames).allTopicNames().get();
- } catch (Exception e) {
- throw new RuntimeException(
- String.format("Failed to get metadata for topics %s.", topicNames), e);
+ AdminClient adminClient, Set<String> topicNames, Properties properties) {
+ int maxRetries =
+ KafkaSourceOptions.getOption(
+ properties,
+ KafkaSourceOptions.TOPIC_METADATA_REQUEST_MAX_RETRY,
+ Integer::parseInt);
+ long retryDelay =
+ KafkaSourceOptions.getOption(
+ properties,
+ KafkaSourceOptions.TOPIC_METADATA_REQUEST_RETRY_INTERVAL_MS,
+ Long::parseLong);
+ for (int attempt = 0; attempt <= maxRetries; attempt++) {
+ try {
+ return adminClient.describeTopics(topicNames).allTopicNames().get();
+ } catch (Exception e) {
+ if (attempt == maxRetries) {
+ throw new RuntimeException(
+ String.format("Failed to get metadata for topics %s.", topicNames), e);
+ } else {
+ LOG.warn(
+ "Attempt {} to get metadata for topics {} failed. Retrying in {} ms...",
+ attempt,
+ topicNames,
+ retryDelay);
+ try {
+ TimeUnit.MILLISECONDS.sleep(retryDelay);
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt(); // Restore interrupted state
+ LOG.error("Thread was interrupted during metadata fetch retry delay.", ie);
+ }
+ }
+ }
Review Comment:
thanks for the suggestion, @AHeise . However, the default value for the `retries` config is already set to MAX_INT (see [code](https://github.com/apache/kafka/blob/3.4/clients/src/main/java/org/apache/kafka/clients/admin/AdminClientConfig.java#L179-L184) and [confluent doc here](https://docs.confluent.io/platform/current/installation/configuration/admin-configs.html#retries). And I believe Flink does not overwrite the config value. w/o my PR, the flink job will fail as soon as the metadata request fails. So I don't think this config control the behavior for failed metadata requests from AdminClient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37366] Allow configurable retry for Kafka topic metadata fetch [flink-connector-kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #155: [FLINK-37366] Allow configurable retry for Kafka topic metadata fetch
URL: https://github.com/apache/flink-connector-kafka/pull/155
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org