You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "showuon (via GitHub)" <gi...@apache.org> on 2023/09/12 09:18:48 UTC
[GitHub] [kafka] showuon opened a new pull request, #14374: MINOR: reduce RLMM retry interval
showuon opened a new pull request, #14374:
URL: https://github.com/apache/kafka/pull/14374
Found this issue while investigating [KAFKA-15453](https://issues.apache.org/jira/browse/KAFKA-15453). The config: `remote.log.metadata.initialization.retry.interval.ms` controls the interval of retrying RemoteLogMetadataManager resources initialization. Default is 5 seconds.
The init failure will happen frequently in scenarios like, 3 brokers start 1 by 1, and when the first 2 brokers start up, it'll fail with RLMM topic creation since no enough brokers to allocation 3 replicas. Then, retrying after 5 seconds. Under these 5 seconds gap, the 2 brokers cannot write/read to/from remote storage, which is not a good UX. It'd better we reduce the default value to make tiered storage feature available ASAP.
This PR reduces the default 5 seconds to 1 second.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on pull request #14374: MINOR: reduce default RLMM retry interval
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on PR #14374:
URL: https://github.com/apache/kafka/pull/14374#issuecomment-1716067232
There are a few unrelated test failures, merging to trunk and 3.6 branches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a diff in pull request #14374: MINOR: reduce default RLMM retry interval
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #14374:
URL: https://github.com/apache/kafka/pull/14374#discussion_r1322951088
##########
storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManagerConfig.java:
##########
@@ -53,7 +53,7 @@ public final class TopicBasedRemoteLogMetadataManagerConfig {
public static final short DEFAULT_REMOTE_LOG_METADATA_TOPIC_REPLICATION_FACTOR = 3;
public static final long DEFAULT_REMOTE_LOG_METADATA_CONSUME_WAIT_MS = 2 * 60 * 1000L;
public static final long DEFAULT_REMOTE_LOG_METADATA_INITIALIZATION_RETRY_MAX_TIMEOUT_MS = 2 * 60 * 1000L;
- public static final long DEFAULT_REMOTE_LOG_METADATA_INITIALIZATION_RETRY_INTERVAL_MS = 5 * 1000L;
+ public static final long DEFAULT_REMOTE_LOG_METADATA_INITIALIZATION_RETRY_INTERVAL_MS = 1000L;
Review Comment:
Sounds good to me. PR updated to 100ms by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #14374: MINOR: reduce RLMM retry interval
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #14374:
URL: https://github.com/apache/kafka/pull/14374#issuecomment-1715337425
@satishd @kamalcph @divijvaidya @clolov , I'd like to hear your thoughts about this change. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd merged pull request #14374: MINOR: reduce default RLMM retry interval
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd merged PR #14374:
URL: https://github.com/apache/kafka/pull/14374
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #14374: MINOR: reduce default RLMM retry interval
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #14374:
URL: https://github.com/apache/kafka/pull/14374#issuecomment-1715352061
Sure, the change makes sense to me. We have an upper bound of these retries defined by
```
public static final long DEFAULT_REMOTE_LOG_METADATA_INITIALIZATION_RETRY_MAX_TIMEOUT_MS = 2 * 60 * 1000L;
```
so I don't think there is a problem with being more eager
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #14374: MINOR: reduce default RLMM retry interval
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14374:
URL: https://github.com/apache/kafka/pull/14374#discussion_r1322945092
##########
storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManagerConfig.java:
##########
@@ -53,7 +53,7 @@ public final class TopicBasedRemoteLogMetadataManagerConfig {
public static final short DEFAULT_REMOTE_LOG_METADATA_TOPIC_REPLICATION_FACTOR = 3;
public static final long DEFAULT_REMOTE_LOG_METADATA_CONSUME_WAIT_MS = 2 * 60 * 1000L;
public static final long DEFAULT_REMOTE_LOG_METADATA_INITIALIZATION_RETRY_MAX_TIMEOUT_MS = 2 * 60 * 1000L;
- public static final long DEFAULT_REMOTE_LOG_METADATA_INITIALIZATION_RETRY_INTERVAL_MS = 5 * 1000L;
+ public static final long DEFAULT_REMOTE_LOG_METADATA_INITIALIZATION_RETRY_INTERVAL_MS = 1000L;
Review Comment:
Nice catch!
We can update this default value now without making any announcement as it is the first release we are releasing these configs for users/devs.
@showuon @kamalcph Should we reduce it to 100ms instead of 1 sec?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kamalcph commented on a diff in pull request #14374: MINOR: reduce default RLMM retry interval
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #14374:
URL: https://github.com/apache/kafka/pull/14374#discussion_r1322763929
##########
storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManagerConfig.java:
##########
@@ -53,7 +53,7 @@ public final class TopicBasedRemoteLogMetadataManagerConfig {
public static final short DEFAULT_REMOTE_LOG_METADATA_TOPIC_REPLICATION_FACTOR = 3;
public static final long DEFAULT_REMOTE_LOG_METADATA_CONSUME_WAIT_MS = 2 * 60 * 1000L;
public static final long DEFAULT_REMOTE_LOG_METADATA_INITIALIZATION_RETRY_MAX_TIMEOUT_MS = 2 * 60 * 1000L;
- public static final long DEFAULT_REMOTE_LOG_METADATA_INITIALIZATION_RETRY_INTERVAL_MS = 5 * 1000L;
+ public static final long DEFAULT_REMOTE_LOG_METADATA_INITIALIZATION_RETRY_INTERVAL_MS = 1000L;
Review Comment:
Nice catch! In our internal branch, it's been configured to 100 ms. All the other configs in TBRLMMC are same as the trunk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kamalcph commented on a diff in pull request #14374: MINOR: reduce default RLMM retry interval
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #14374:
URL: https://github.com/apache/kafka/pull/14374#discussion_r1322763929
##########
storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManagerConfig.java:
##########
@@ -53,7 +53,7 @@ public final class TopicBasedRemoteLogMetadataManagerConfig {
public static final short DEFAULT_REMOTE_LOG_METADATA_TOPIC_REPLICATION_FACTOR = 3;
public static final long DEFAULT_REMOTE_LOG_METADATA_CONSUME_WAIT_MS = 2 * 60 * 1000L;
public static final long DEFAULT_REMOTE_LOG_METADATA_INITIALIZATION_RETRY_MAX_TIMEOUT_MS = 2 * 60 * 1000L;
- public static final long DEFAULT_REMOTE_LOG_METADATA_INITIALIZATION_RETRY_INTERVAL_MS = 5 * 1000L;
+ public static final long DEFAULT_REMOTE_LOG_METADATA_INITIALIZATION_RETRY_INTERVAL_MS = 1000L;
Review Comment:
Nice catch! In our internal branch, it's been configured to 100 ms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org