You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "majialoong (via GitHub)" <gi...@apache.org> on 2026/03/23 16:22:34 UTC
[PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
majialoong opened a new pull request, #21855:
URL: https://github.com/apache/kafka/pull/21855
`createGroupConfigEntry` used the group config name (e.g. `consumer.session.timeout.ms`) directly
to look up broker synonyms, which could produce incorrect synonym chains.
This change prepends `GROUP_PREFIX` to derive the correct broker config name
(e.g. `group.consumer.session.timeout.ms`) for synonym lookup, and validates with
`configKeys().containsKey()` to fall back to `GroupConfig` defaults for configs without
broker-level counterparts (e.g. `share.auto.offset.reset`).
It also reads current broker config values via `KafkaConfig.get()` instead of startup-frozen
defaults, so that dynamically updated broker configs are correctly reflected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3006407487
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupConfigTest.java:
##########
@@ -637,6 +640,55 @@ public void testEvaluateMinBoundedValueBelowMinIsCapped(
assertEquals(expectedMin, result.get(key));
}
+ @Test
+ public void testAllGroupConfigSynonyms() {
+ // GroupConfig entries with no broker-level synonym.
+ Set<String> groupConfigsWithoutBrokerSynonym = Set.of(
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ GroupConfig.SHARE_ISOLATION_LEVEL_CONFIG,
+ GroupConfig.SHARE_RENEW_ACKNOWLEDGE_ENABLE_CONFIG
+ );
Review Comment:
I would suggest a different approach. Should we make `ALL_GROUP_CONFIG_SYNONYMS` a `Map<String, Optional<String>>` and ensure that all configs have an entry in the map? Those three could use `Optional.empty()` to signal that they have no synonyms.
##########
core/src/test/scala/unit/kafka/server/KafkaApisTest.scala:
##########
@@ -405,6 +405,102 @@ class KafkaApisTest extends Logging {
assertEquals(cgConfigs.size, configs.size)
}
+ @Test
+ def testDescribeConfigsGroupSynonyms(): Unit = {
Review Comment:
Do we cover all the cases from your comment? It may be worth splitting it into smaller test cases.
##########
core/src/test/scala/unit/kafka/server/KafkaApisTest.scala:
##########
@@ -405,6 +405,102 @@ class KafkaApisTest extends Logging {
assertEquals(cgConfigs.size, configs.size)
}
+ @Test
+ def testDescribeConfigsGroupSynonyms(): Unit = {
Review Comment:
We could also add them as integration tests. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3010657294
##########
core/src/test/scala/integration/kafka/api/BaseAdminIntegrationTest.scala:
##########
@@ -202,6 +202,13 @@ abstract class BaseAdminIntegrationTest extends IntegrationTestHarness with Logg
config.setProperty(ServerLogConfigs.LOG_ROLL_TIME_JITTER_MILLIS_CONFIG, "123")
})
}
+ // For testDescribeGroupConfigSynonymsWithStaticBrokerConfig, set a static broker config so
+ // that we can verify the synonym chain includes STATIC_BROKER_CONFIG.
+ if (testInfo.getTestMethod.toString.contains("testDescribeGroupConfigSynonymsWithStaticBrokerConfig")) {
+ configs.foreach(config => {
+ config.setProperty(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "2000")
+ })
+ }
Review Comment:
Could we rewrite these tests using the new test infra? It makes setting static configs much easier
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3000284317
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorConfig.java:
##########
@@ -392,6 +392,8 @@ public class GroupCoordinatorConfig {
STREAMS_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
STREAMS_GROUP_ASSIGNOR_OFFLOAD_ENABLE_CONFIG
);
+
+ public static final String GROUP_PREFIX = "group.";
Review Comment:
I am also a bit on the fence here. I think that having an explicit mapping is better and less error prone. We could have `ALL_GROUP_CONFIG_SYNONYMS`, as suggested by @chia7712, in `GroupConfigs`. I would also add a test to ensure that all configs have an entry in the Map to ensure that future configs are considered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3001207378
##########
core/src/main/scala/kafka/server/ConfigHelper.scala:
##########
@@ -161,15 +161,22 @@ class ConfigHelper(metadataCache: MetadataCache, config: KafkaConfig, configRepo
private def createGroupConfigEntry(groupConfig: GroupConfig, groupProps: Properties, includeSynonyms: Boolean, includeDocumentation: Boolean)
(name: String, value: Any): DescribeConfigsResponseData.DescribeConfigsResourceResult = {
- val allNames = brokerSynonyms(name)
val configEntryType = GroupConfig.configType(name).toScala
val isSensitive = KafkaConfig.maybeSensitive(configEntryType)
val valueAsString = if (isSensitive) null else ConfigDef.convertToString(value, configEntryType.orNull)
val allSynonyms = {
- val list = configSynonyms(name, allNames, isSensitive)
+ val brokerConfigName = GroupCoordinatorConfig.GROUP_PREFIX + name
+ val list = if (KafkaConfig.configDef.configKeys().containsKey(brokerConfigName))
Review Comment:
nit: Should we just remove `configDef()` and make `CONFIG` public?
##########
core/src/main/scala/kafka/server/ConfigHelper.scala:
##########
@@ -161,15 +161,22 @@ class ConfigHelper(metadataCache: MetadataCache, config: KafkaConfig, configRepo
private def createGroupConfigEntry(groupConfig: GroupConfig, groupProps: Properties, includeSynonyms: Boolean, includeDocumentation: Boolean)
(name: String, value: Any): DescribeConfigsResponseData.DescribeConfigsResourceResult = {
- val allNames = brokerSynonyms(name)
val configEntryType = GroupConfig.configType(name).toScala
val isSensitive = KafkaConfig.maybeSensitive(configEntryType)
val valueAsString = if (isSensitive) null else ConfigDef.convertToString(value, configEntryType.orNull)
val allSynonyms = {
- val list = configSynonyms(name, allNames, isSensitive)
+ val brokerConfigName = GroupCoordinatorConfig.GROUP_PREFIX + name
+ val list = if (KafkaConfig.configDef.configKeys().containsKey(brokerConfigName))
+ configSynonyms(brokerConfigName, brokerSynonyms(brokerConfigName), isSensitive)
+ else
+ // No broker synonym, fall back to GroupConfig defaults
+ Option(GroupConfig.configDef().defaultValues().get(name))
+ .map(v => List(new DescribeConfigsResponseData.DescribeConfigsSynonym().setName(name)
Review Comment:
nit: Should we put `.setName` on new line too?
##########
core/src/main/scala/kafka/server/ConfigHelper.scala:
##########
@@ -304,4 +311,17 @@ class ConfigHelper(metadataCache: MetadataCache, config: KafkaConfig, configRepo
throw new InvalidRequestException(s"Broker id must be an integer, but it is: $resourceName")
}
}
+
+ /**
+ * Build the current effective group config defaults by reading from KafkaConfig.
+ */
+ private def currentGroupConfigDefaults: java.util.Map[String, Object] = {
Review Comment:
I wonder if we should rather place this one in `KafkaConfig`. What do you think?
##########
core/src/main/scala/kafka/server/ConfigHelper.scala:
##########
@@ -161,15 +161,22 @@ class ConfigHelper(metadataCache: MetadataCache, config: KafkaConfig, configRepo
private def createGroupConfigEntry(groupConfig: GroupConfig, groupProps: Properties, includeSynonyms: Boolean, includeDocumentation: Boolean)
(name: String, value: Any): DescribeConfigsResponseData.DescribeConfigsResourceResult = {
- val allNames = brokerSynonyms(name)
val configEntryType = GroupConfig.configType(name).toScala
val isSensitive = KafkaConfig.maybeSensitive(configEntryType)
val valueAsString = if (isSensitive) null else ConfigDef.convertToString(value, configEntryType.orNull)
val allSynonyms = {
- val list = configSynonyms(name, allNames, isSensitive)
+ val brokerConfigName = GroupCoordinatorConfig.GROUP_PREFIX + name
+ val list = if (KafkaConfig.configDef.configKeys().containsKey(brokerConfigName))
Review Comment:
While we are here, would it be possible to not return internal configs? We don't have any at the moment but it would be good to cover it as may have one coming soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3011252255
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupConfig.java:
##########
@@ -281,8 +281,41 @@ public final class GroupConfig extends AbstractConfig {
MEDIUM,
GroupCoordinatorConfig.STREAMS_GROUP_TASK_OFFSET_INTERVAL_MS_DOC);
+ /**
+ * Mapping from GroupConfig name to its broker-level synonym config name.
+ * {@code Optional.empty()} indicates that the config has no broker-level synonym.
+ */
+ public static final Map<String, Optional<String>> ALL_GROUP_CONFIG_SYNONYMS = Map.ofEntries(
Review Comment:
Good suggestion. I checked, but `Map.of() `only supports up to 10 key-value pairs. Since `ALL_GROUP_CONFIG_SYNONYMS` has more than 10 entries, `Map.ofEntries()` is the appropriate option here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3011368584
##########
core/src/test/scala/integration/kafka/api/PlaintextAdminIntegrationTest.scala:
##########
@@ -1203,6 +1203,221 @@ class PlaintextAdminIntegrationTest extends BaseAdminIntegrationTest {
assertEquals(50000, brokerServers.head.groupConfigManager.groupConfig(groupId).get.consumerSessionTimeoutMs)
}
+ @Test
+ def testDescribeGroupConfigSynonymsWithBrokerSynonym(): Unit = {
Review Comment:
Great suggestion. I’ve updated the tests accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on PR #21855:
URL: https://github.com/apache/kafka/pull/21855#issuecomment-4111943016
Below is the output of `kafka-configs.sh --describe` for a consumer group after the fix:
```
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --describe --entity-name my-group --all
All configs for group my-group are:
consumer.assignment.interval.ms=1555 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.assignment.interval.ms=1555, DYNAMIC_BROKER_CONFIG:group.consumer.assignment.interval.ms=1333, STATIC_BROKER_CONFIG:group.consumer.assignment.interval.ms=1111, DEFAULT_CONFIG:group.consumer.assignment.interval.ms=1000}
consumer.assignor.offload.enable=true sensitive=false synonyms={DEFAULT_CONFIG:group.consumer.assignor.offload.enable=true}
consumer.heartbeat.interval.ms=5555 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.heartbeat.interval.ms=5555, DEFAULT_CONFIG:group.consumer.heartbeat.interval.ms=5000}
consumer.session.timeout.ms=46000 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.session.timeout.ms=46000, STATIC_BROKER_CONFIG:group.consumer.session.timeout.ms=47000, DEFAULT_CONFIG:group.consumer.session.timeout.ms=45000}
share.assignment.interval.ms=1999 sensitive=false synonyms={STATIC_BROKER_CONFIG:group.share.assignment.interval.ms=1999, DEFAULT_CONFIG:group.share.assignment.interval.ms=1000}
share.assignor.offload.enable=true sensitive=false synonyms={DEFAULT_CONFIG:group.share.assignor.offload.enable=true}
share.auto.offset.reset=earliest sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:share.auto.offset.reset=earliest, DEFAULT_CONFIG:share.auto.offset.reset=latest}
share.delivery.count.limit=5 sensitive=false synonyms={DEFAULT_CONFIG:group.share.delivery.count.limit=5}
share.heartbeat.interval.ms=5000 sensitive=false synonyms={DEFAULT_CONFIG:group.share.heartbeat.interval.ms=5000}
share.isolation.level=read_uncommitted sensitive=false synonyms={DEFAULT_CONFIG:share.isolation.level=read_uncommitted}
share.partition.max.record.locks=2000 sensitive=false synonyms={DEFAULT_CONFIG:group.share.partition.max.record.locks=2000}
share.record.lock.duration.ms=30000 sensitive=false synonyms={DEFAULT_CONFIG:group.share.record.lock.duration.ms=30000}
share.renew.acknowledge.enable=true sensitive=false synonyms={DEFAULT_CONFIG:share.renew.acknowledge.enable=true}
share.session.timeout.ms=45000 sensitive=false synonyms={DEFAULT_CONFIG:group.share.session.timeout.ms=45000}
streams.assignment.interval.ms=1000 sensitive=false synonyms={DEFAULT_CONFIG:group.streams.assignment.interval.ms=1000}
streams.assignor.offload.enable=true sensitive=false synonyms={DEFAULT_CONFIG:group.streams.assignor.offload.enable=true}
streams.heartbeat.interval.ms=5000 sensitive=false synonyms={DEFAULT_CONFIG:group.streams.heartbeat.interval.ms=5000}
streams.initial.rebalance.delay.ms=3000 sensitive=false synonyms={DEFAULT_CONFIG:group.streams.initial.rebalance.delay.ms=3000}
streams.num.standby.replicas=0 sensitive=false synonyms={DEFAULT_CONFIG:group.streams.num.standby.replicas=0}
streams.session.timeout.ms=45000 sensitive=false synonyms={DEFAULT_CONFIG:group.streams.session.timeout.ms=45000}
streams.task.offset.interval.ms=60000 sensitive=false synonyms={DEFAULT_CONFIG:group.streams.task.offset.interval.ms=60000}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3010301867
##########
core/src/main/scala/kafka/server/ConfigHelper.scala:
##########
@@ -161,15 +161,23 @@ class ConfigHelper(metadataCache: MetadataCache, config: KafkaConfig, configRepo
private def createGroupConfigEntry(groupConfig: GroupConfig, groupProps: Properties, includeSynonyms: Boolean, includeDocumentation: Boolean)
(name: String, value: Any): DescribeConfigsResponseData.DescribeConfigsResourceResult = {
- val allNames = brokerSynonyms(name)
val configEntryType = GroupConfig.configType(name).toScala
val isSensitive = KafkaConfig.maybeSensitive(configEntryType)
val valueAsString = if (isSensitive) null else ConfigDef.convertToString(value, configEntryType.orNull)
val allSynonyms = {
- val list = configSynonyms(name, allNames, isSensitive)
+ val brokerConfigName = GroupConfig.ALL_GROUP_CONFIG_SYNONYMS.get(name)
+ val list = if (brokerConfigName != null)
+ configSynonyms(brokerConfigName, brokerSynonyms(brokerConfigName), isSensitive)
+ else
+ // No broker synonym, fall back to GroupConfig defaults
+ Option(GroupConfig.CONFIG_DEF.defaultValues().get(name))
Review Comment:
Applying the fallback to topic also makes sense. I’ll follow up with a separate JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac merged PR #21855:
URL: https://github.com/apache/kafka/pull/21855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3002017511
##########
core/src/main/scala/kafka/server/ConfigHelper.scala:
##########
@@ -304,4 +311,17 @@ class ConfigHelper(metadataCache: MetadataCache, config: KafkaConfig, configRepo
throw new InvalidRequestException(s"Broker id must be an integer, but it is: $resourceName")
}
}
+
+ /**
+ * Build the current effective group config defaults by reading from KafkaConfig.
+ */
+ private def currentGroupConfigDefaults: java.util.Map[String, Object] = {
Review Comment:
Makes sense. I'll move it to `KafkaConfig` and name it `extractGroupConfigMap` to be consistent with `extractLogConfigMap`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3002049137
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorConfig.java:
##########
@@ -392,6 +392,8 @@ public class GroupCoordinatorConfig {
STREAMS_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
STREAMS_GROUP_ASSIGNOR_OFFLOAD_ENABLE_CONFIG
);
+
+ public static final String GROUP_PREFIX = "group.";
Review Comment:
I’ll introduce an explicit mapping for this and add a test to ensure all group configs are covered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3010296970
##########
core/src/main/scala/kafka/server/KafkaConfig.scala:
##########
@@ -607,4 +607,12 @@ class KafkaConfig private(doLog: Boolean, val props: util.Map[_, _])
logProps.put(TopicConfig.LOCAL_LOG_RETENTION_BYTES_CONFIG, remoteLogManagerConfig.logLocalRetentionBytes: java.lang.Long)
logProps
}
+
+ def extractGroupConfigMap: java.util.Map[String, Object] = {
Review Comment:
Agreed. I’ve moved this to `AbstractKafkaConfig`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3001981793
##########
core/src/main/scala/kafka/server/ConfigHelper.scala:
##########
@@ -161,15 +161,22 @@ class ConfigHelper(metadataCache: MetadataCache, config: KafkaConfig, configRepo
private def createGroupConfigEntry(groupConfig: GroupConfig, groupProps: Properties, includeSynonyms: Boolean, includeDocumentation: Boolean)
(name: String, value: Any): DescribeConfigsResponseData.DescribeConfigsResourceResult = {
- val allNames = brokerSynonyms(name)
val configEntryType = GroupConfig.configType(name).toScala
val isSensitive = KafkaConfig.maybeSensitive(configEntryType)
val valueAsString = if (isSensitive) null else ConfigDef.convertToString(value, configEntryType.orNull)
val allSynonyms = {
- val list = configSynonyms(name, allNames, isSensitive)
+ val brokerConfigName = GroupCoordinatorConfig.GROUP_PREFIX + name
+ val list = if (KafkaConfig.configDef.configKeys().containsKey(brokerConfigName))
Review Comment:
Just to confirm, is it `GroupConfig.configDef()` you'd like removed (making `CONFIG` public)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3007182419
##########
core/src/test/scala/unit/kafka/server/KafkaApisTest.scala:
##########
@@ -405,6 +405,102 @@ class KafkaApisTest extends Logging {
assertEquals(cgConfigs.size, configs.size)
}
+ @Test
+ def testDescribeConfigsGroupSynonyms(): Unit = {
Review Comment:
Makes sense. I'll split it into smaller test cases for each scenario and add corresponding integration tests as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3013868324
##########
core/src/main/scala/kafka/server/ConfigHelper.scala:
##########
@@ -161,15 +161,23 @@ class ConfigHelper(metadataCache: MetadataCache, config: KafkaConfig, configRepo
private def createGroupConfigEntry(groupConfig: GroupConfig, groupProps: Properties, includeSynonyms: Boolean, includeDocumentation: Boolean)
(name: String, value: Any): DescribeConfigsResponseData.DescribeConfigsResourceResult = {
- val allNames = brokerSynonyms(name)
val configEntryType = GroupConfig.configType(name).toScala
val isSensitive = KafkaConfig.maybeSensitive(configEntryType)
val valueAsString = if (isSensitive) null else ConfigDef.convertToString(value, configEntryType.orNull)
val allSynonyms = {
- val list = configSynonyms(name, allNames, isSensitive)
+ val brokerConfigName = GroupConfig.ALL_GROUP_CONFIG_SYNONYMS.get(name)
+ val list = if (brokerConfigName != null)
+ configSynonyms(brokerConfigName, brokerSynonyms(brokerConfigName), isSensitive)
+ else
+ // No broker synonym, fall back to GroupConfig defaults
+ Option(GroupConfig.CONFIG_DEF.defaultValues().get(name))
Review Comment:
I opened KAFKA-20374 to track the topic fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on PR #21855:
URL: https://github.com/apache/kafka/pull/21855#issuecomment-4171056792
@dajac Thanks for the review! I've addressed all the suggestions. Feel free to let me know if you have any further feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on PR #21855:
URL: https://github.com/apache/kafka/pull/21855#issuecomment-4118705525
> @majialoong Thanks for the patch! Could you please attach the output before the patch too?
@dajac Thanks for the review. Here are the before/after comparisons for `kafka-configs.sh --describe` across different scenarios, using `consumer.assignment.interval.ms` as the example config:
1. Invalid broker param in `server.properties`
Adding `consumer.assignment.interval.ms=1111` (missing group. prefix) to `server.properties`.
Before:
```bash
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
consumer.assignment.interval.ms=1000 sensitive=false synonyms={DEFAULT_CONFIG:consumer.assignment.interval.ms=1000, STATIC_BROKER_CONFIG:consumer.assignment.interval.ms=1111}
...
```
After:
```bash
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
consumer.assignment.interval.ms=1000 sensitive=false synonyms={DEFAULT_CONFIG:group.consumer.assignment.interval.ms=1000}
...
```
The invalid config in `server.properties` no longer produces a spurious `STATIC_BROKER_CONFIG` synonym.
---
2. Valid broker param in `server.properties`
Adding `group.consumer.assignment.interval.ms=1222` to `server.properties`.
Before:
```bash
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
consumer.assignment.interval.ms=1000 sensitive=false synonyms={DEFAULT_CONFIG:consumer.assignment.interval.ms=1000}
...
```
After:
```bash
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
consumer.assignment.interval.ms=1222 sensitive=false synonyms={STATIC_BROKER_CONFIG:group.consumer.assignment.interval.ms=1222, DEFAULT_CONFIG:group.consumer.assignment.interval.ms=1000}
...
```
The broker override is now correctly reflected in both the displayed value and the synonym chain.
---
3. Group-level override only
No `server.properties` changes. Setting group config via:
`bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type groups --entity-name test-group --add-config consumer.assignment.interval.ms=1333`
Before:
```bash
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe -all
All configs for group test-group are:
consumer.assignment.interval.ms=1333 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.assignment.interval.ms=1333}
...
```
After:
```bash
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
consumer.assignment.interval.ms=1333 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.assignment.interval.ms=1333, DEFAULT_CONFIG:group.consumer.assignment.interval.ms=1000}
...
```
The `DEFAULT_CONFIG` synonym with the correct broker config name is now present.
---
4. Valid broker param + group-level override
Adding `group.consumer.assignment.interval.ms=1222` to `server.properties`,
then setting group config via:
`bin/kafka-configs.sh --alter --entity-type groups --entity-name test-group --add-config consumer.assignment.interval.ms=1333`
Before:
```bash
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
consumer.assignment.interval.ms=1333 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.assignment.interval.ms=1333}
...
```
After:
```bash
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
consumer.assignment.interval.ms=1333 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.assignment.interval.ms=1333, STATIC_BROKER_CONFIG:group.consumer.assignment.interval.ms=1222, DEFAULT_CONFIG:group.consumer.assignment.interval.ms=1000}
...
```
The full synonym chain `DYNAMIC_GROUP_CONFIG → STATIC_BROKER_CONFIG → DEFAULT_CONFIG` is now displayed correctly.
---
5. Valid broker param + group-level override + dynamic broker config
Adding `group.consumer.assignment.interval.ms=1222` `to server.properties`,
then:
`bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type brokers --entity-default --add-config group.consumer.assignment.interval.ms=1444`
`bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type groups --entity-name test-group --add-config consumer.assignment.interval.ms=1333`
Before:
```bash
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
consumer.assignment.interval.ms=1333 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.assignment.interval.ms=1333}
...
```
After:
```bash
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
consumer.assignment.interval.ms=1333 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.assignment.interval.ms=1333, DYNAMIC_DEFAULT_BROKER_CONFIG:group.consumer.assignment.interval.ms=1444, STATIC_BROKER_CONFIG:group.consumer.assignment.interval.ms=1222, DEFAULT_CONFIG:group.consumer.assignment.interval.ms=1000}
...
```
The complete synonym chain `DYNAMIC_GROUP_CONFIG → DYNAMIC_DEFAULT_BROKER_CONFIG → STATIC_BROKER_CONFIG → DEFAULT_CONFIG `is now displayed correctly.
---
6. Config without broker synonym (`share.auto.offset.reset`)
Setting group config via:
`bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type groups --entity-name test-group --add-config share.auto.offset.reset=latest`
Before:
```bash
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
...
share.auto.offset.reset=latest sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:share.auto.offset.reset=latest}
...
```
After:
```bash
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
...
share.auto.offset.reset=latest sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:share.auto.offset.reset=latest, DEFAULT_CONFIG:share.auto.offset.reset=latest}
...
```
Configs without a broker-level synonym correctly fall back to GroupConfig defaults.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #21855:
URL: https://github.com/apache/kafka/pull/21855#issuecomment-4172290183
Merged to trunk and 4.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3010571154
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupConfig.java:
##########
@@ -281,8 +281,41 @@ public final class GroupConfig extends AbstractConfig {
MEDIUM,
GroupCoordinatorConfig.STREAMS_GROUP_TASK_OFFSET_INTERVAL_MS_DOC);
+ /**
+ * Mapping from GroupConfig name to its broker-level synonym config name.
+ * {@code Optional.empty()} indicates that the config has no broker-level synonym.
+ */
+ public static final Map<String, Optional<String>> ALL_GROUP_CONFIG_SYNONYMS = Map.ofEntries(
Review Comment:
Can't we use `Map.of`?
##########
core/src/test/scala/integration/kafka/api/BaseAdminIntegrationTest.scala:
##########
@@ -202,6 +202,13 @@ abstract class BaseAdminIntegrationTest extends IntegrationTestHarness with Logg
config.setProperty(ServerLogConfigs.LOG_ROLL_TIME_JITTER_MILLIS_CONFIG, "123")
})
}
+ // For testDescribeGroupConfigSynonymsWithStaticBrokerConfig, set a static broker config so
+ // that we can verify the synonym chain includes STATIC_BROKER_CONFIG.
+ if (testInfo.getTestMethod.toString.contains("testDescribeGroupConfigSynonymsWithStaticBrokerConfig")) {
+ configs.foreach(config => {
+ config.setProperty(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "2000")
+ })
+ }
Review Comment:
Ouch. This is not your fault but this is pretty ugly.
##########
core/src/test/scala/integration/kafka/api/PlaintextAdminIntegrationTest.scala:
##########
@@ -1203,6 +1203,221 @@ class PlaintextAdminIntegrationTest extends BaseAdminIntegrationTest {
assertEquals(50000, brokerServers.head.groupConfigManager.groupConfig(groupId).get.consumerSessionTimeoutMs)
}
+ @Test
+ def testDescribeGroupConfigSynonymsWithBrokerSynonym(): Unit = {
Review Comment:
Have you considered creating a new suite in `kafka/clients/clients-integration-tests`, similar to `StaticBrokerConfigTest` for those new tests (e.g. `DynamicGroupConfigIntegrationTest`? This would allow you to use `@ClusterTest` so you can directly set the broker configs that you need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #21855:
URL: https://github.com/apache/kafka/pull/21855#issuecomment-4145003870
> > @majialoong Thanks for all the cases! Do we have test cases for each of them?
>
> @dajac The current `testDescribeConfigsGroupSynonyms` covers some scenarios. I'll add more test cases for the remaining ones and split them into separate methods so each test focuses on a single synonym chain pattern.
Sounds good. I wonder whether we could add integration tests too. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3007384898
##########
core/src/main/scala/kafka/server/ConfigHelper.scala:
##########
@@ -161,15 +161,23 @@ class ConfigHelper(metadataCache: MetadataCache, config: KafkaConfig, configRepo
private def createGroupConfigEntry(groupConfig: GroupConfig, groupProps: Properties, includeSynonyms: Boolean, includeDocumentation: Boolean)
(name: String, value: Any): DescribeConfigsResponseData.DescribeConfigsResourceResult = {
- val allNames = brokerSynonyms(name)
val configEntryType = GroupConfig.configType(name).toScala
val isSensitive = KafkaConfig.maybeSensitive(configEntryType)
val valueAsString = if (isSensitive) null else ConfigDef.convertToString(value, configEntryType.orNull)
val allSynonyms = {
- val list = configSynonyms(name, allNames, isSensitive)
+ val brokerConfigName = GroupConfig.ALL_GROUP_CONFIG_SYNONYMS.get(name)
+ val list = if (brokerConfigName != null)
+ configSynonyms(brokerConfigName, brokerSynonyms(brokerConfigName), isSensitive)
+ else
+ // No broker synonym, fall back to GroupConfig defaults
+ Option(GroupConfig.CONFIG_DEF.defaultValues().get(name))
Review Comment:
Should we apply this fallback to topic?
##########
core/src/main/scala/kafka/server/KafkaConfig.scala:
##########
@@ -607,4 +607,12 @@ class KafkaConfig private(doLog: Boolean, val props: util.Map[_, _])
logProps.put(TopicConfig.LOCAL_LOG_RETENTION_BYTES_CONFIG, remoteLogManagerConfig.logLocalRetentionBytes: java.lang.Long)
logProps
}
+
+ def extractGroupConfigMap: java.util.Map[String, Object] = {
Review Comment:
Could you please move this to `AbstractKafkaConfig`? We are trying to phase out `KafkaConfig`, so it would be better to avoid adding new code to it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3007189344
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupConfigTest.java:
##########
@@ -637,6 +640,55 @@ public void testEvaluateMinBoundedValueBelowMinIsCapped(
assertEquals(expectedMin, result.get(key));
}
+ @Test
+ public void testAllGroupConfigSynonyms() {
+ // GroupConfig entries with no broker-level synonym.
+ Set<String> groupConfigsWithoutBrokerSynonym = Set.of(
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ GroupConfig.SHARE_ISOLATION_LEVEL_CONFIG,
+ GroupConfig.SHARE_RENEW_ACKNOWLEDGE_ENABLE_CONFIG
+ );
Review Comment:
Agreed, I'll update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3022868525
##########
core/src/main/scala/kafka/server/ConfigHelper.scala:
##########
@@ -161,15 +161,23 @@ class ConfigHelper(metadataCache: MetadataCache, config: KafkaConfig, configRepo
private def createGroupConfigEntry(groupConfig: GroupConfig, groupProps: Properties, includeSynonyms: Boolean, includeDocumentation: Boolean)
(name: String, value: Any): DescribeConfigsResponseData.DescribeConfigsResourceResult = {
- val allNames = brokerSynonyms(name)
val configEntryType = GroupConfig.configType(name).toScala
val isSensitive = KafkaConfig.maybeSensitive(configEntryType)
val valueAsString = if (isSensitive) null else ConfigDef.convertToString(value, configEntryType.orNull)
val allSynonyms = {
- val list = configSynonyms(name, allNames, isSensitive)
+ val brokerConfigName = GroupConfig.ALL_GROUP_CONFIG_SYNONYMS.get(name)
Review Comment:
Good idea! I used `brokerSynonym` to make the mapping direction explicit. Happy to switch to `synonym` if you prefer just let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #21855:
URL: https://github.com/apache/kafka/pull/21855#issuecomment-4112740924
@majialoong Thanks for the patch! Could you please attach the output before the patch too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3010662119
##########
core/src/test/scala/integration/kafka/api/PlaintextAdminIntegrationTest.scala:
##########
@@ -1203,6 +1203,221 @@ class PlaintextAdminIntegrationTest extends BaseAdminIntegrationTest {
assertEquals(50000, brokerServers.head.groupConfigManager.groupConfig(groupId).get.consumerSessionTimeoutMs)
}
+ @Test
+ def testDescribeGroupConfigSynonymsWithBrokerSynonym(): Unit = {
Review Comment:
+100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r2978506510
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorConfig.java:
##########
@@ -392,6 +392,8 @@ public class GroupCoordinatorConfig {
STREAMS_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
STREAMS_GROUP_ASSIGNOR_OFFLOAD_ENABLE_CONFIG
);
+
+ public static final String GROUP_PREFIX = "group.";
Review Comment:
I'm a bit concerned about using an implicit string prefix to connect to server-level configurations. It can be error-prone and less flexible. Could we follow the pattern of `ALL_TOPIC_CONFIG_SYNONYMS` and introduce something like `ALL_GROUP_CONFIG_SYNONYMS`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3017098395
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/admin/DynamicGroupConfigIntegrationTest.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.test.ClusterInstance;
+import org.apache.kafka.common.test.api.ClusterConfigProperty;
+import org.apache.kafka.common.test.api.ClusterTest;
+import org.apache.kafka.common.test.api.Type;
+import org.apache.kafka.coordinator.group.GroupConfig;
+import org.apache.kafka.coordinator.group.GroupCoordinatorConfig;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class DynamicGroupConfigIntegrationTest {
+
+ @ClusterTest(types = {Type.KRAFT})
+ public void testDescribeGroupConfigSynonymsWithBrokerSynonym(ClusterInstance cluster) throws Exception {
+ try (Admin admin = cluster.admin()) {
+ String group = "synonym-test-group";
Review Comment:
nit: It seems that we could use `var` to reduce the verbosity of the code.
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/admin/DynamicGroupConfigIntegrationTest.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.test.ClusterInstance;
+import org.apache.kafka.common.test.api.ClusterConfigProperty;
+import org.apache.kafka.common.test.api.ClusterTest;
+import org.apache.kafka.common.test.api.Type;
+import org.apache.kafka.coordinator.group.GroupConfig;
+import org.apache.kafka.coordinator.group.GroupCoordinatorConfig;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class DynamicGroupConfigIntegrationTest {
+
+ @ClusterTest(types = {Type.KRAFT})
+ public void testDescribeGroupConfigSynonymsWithBrokerSynonym(ClusterInstance cluster) throws Exception {
+ try (Admin admin = cluster.admin()) {
+ String group = "synonym-test-group";
+ ConfigResource groupResource = new ConfigResource(ConfigResource.Type.GROUP, group);
+ ConfigResource brokerResource = new ConfigResource(ConfigResource.Type.BROKER, "0");
+ ConfigResource brokerDefaultResource = new ConfigResource(ConfigResource.Type.BROKER, "");
+
+ // Verify default config only.
+ // Expected synonym chain: DEFAULT_CONFIG
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ String.valueOf(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_DEFAULT),
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set per-broker dynamic config.
+ // Expected synonym chain: DYNAMIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "1500"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "1500",
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set dynamic default broker config; per-broker config still takes precedence.
+ // Expected synonym chain: DYNAMIC_BROKER_CONFIG -> DYNAMIC_DEFAULT_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerDefaultResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "2000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "1500",
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set group override; it takes precedence over all broker configs.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DYNAMIC_BROKER_CONFIG
+ // -> DYNAMIC_DEFAULT_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG, "3000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "3000",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+ }
+ }
+
+ @ClusterTest(types = {Type.KRAFT})
+ public void testDescribeGroupConfigSynonymsWithoutBrokerSynonym(ClusterInstance cluster) throws Exception {
+ try (Admin admin = cluster.admin()) {
+ String group = "synonym-no-broker-test-group";
+ ConfigResource groupResource = new ConfigResource(ConfigResource.Type.GROUP, group);
+
+ // Verify default config for a config with no broker synonym.
+ // Expected synonym chain: DEFAULT_CONFIG
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_DEFAULT,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set group override; synonyms use group config name since there is no broker synonym.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG, "earliest"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
Review Comment:
This code is repeated many times. I wonder whether we could have an helper method to update group configs or default broker configs.
##########
core/src/main/scala/kafka/server/ConfigHelper.scala:
##########
@@ -161,15 +161,23 @@ class ConfigHelper(metadataCache: MetadataCache, config: KafkaConfig, configRepo
private def createGroupConfigEntry(groupConfig: GroupConfig, groupProps: Properties, includeSynonyms: Boolean, includeDocumentation: Boolean)
(name: String, value: Any): DescribeConfigsResponseData.DescribeConfigsResourceResult = {
- val allNames = brokerSynonyms(name)
val configEntryType = GroupConfig.configType(name).toScala
val isSensitive = KafkaConfig.maybeSensitive(configEntryType)
val valueAsString = if (isSensitive) null else ConfigDef.convertToString(value, configEntryType.orNull)
val allSynonyms = {
- val list = configSynonyms(name, allNames, isSensitive)
+ val brokerConfigName = GroupConfig.ALL_GROUP_CONFIG_SYNONYMS.get(name)
+ val list = if (brokerConfigName != null && brokerConfigName.isPresent)
+ configSynonyms(brokerConfigName.get, brokerSynonyms(brokerConfigName.get), isSensitive)
+ else
+ // No broker synonym, fall back to GroupConfig defaults
+ Option(GroupConfig.CONFIG_DEF.defaultValues().get(name))
Review Comment:
ditto.
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/admin/DynamicGroupConfigIntegrationTest.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.test.ClusterInstance;
+import org.apache.kafka.common.test.api.ClusterConfigProperty;
+import org.apache.kafka.common.test.api.ClusterTest;
+import org.apache.kafka.common.test.api.Type;
+import org.apache.kafka.coordinator.group.GroupConfig;
+import org.apache.kafka.coordinator.group.GroupCoordinatorConfig;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class DynamicGroupConfigIntegrationTest {
+
+ @ClusterTest(types = {Type.KRAFT})
+ public void testDescribeGroupConfigSynonymsWithBrokerSynonym(ClusterInstance cluster) throws Exception {
+ try (Admin admin = cluster.admin()) {
+ String group = "synonym-test-group";
+ ConfigResource groupResource = new ConfigResource(ConfigResource.Type.GROUP, group);
+ ConfigResource brokerResource = new ConfigResource(ConfigResource.Type.BROKER, "0");
+ ConfigResource brokerDefaultResource = new ConfigResource(ConfigResource.Type.BROKER, "");
+
+ // Verify default config only.
+ // Expected synonym chain: DEFAULT_CONFIG
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ String.valueOf(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_DEFAULT),
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set per-broker dynamic config.
+ // Expected synonym chain: DYNAMIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "1500"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "1500",
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set dynamic default broker config; per-broker config still takes precedence.
+ // Expected synonym chain: DYNAMIC_BROKER_CONFIG -> DYNAMIC_DEFAULT_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerDefaultResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "2000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "1500",
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set group override; it takes precedence over all broker configs.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DYNAMIC_BROKER_CONFIG
+ // -> DYNAMIC_DEFAULT_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG, "3000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "3000",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+ }
+ }
+
+ @ClusterTest(types = {Type.KRAFT})
+ public void testDescribeGroupConfigSynonymsWithoutBrokerSynonym(ClusterInstance cluster) throws Exception {
+ try (Admin admin = cluster.admin()) {
+ String group = "synonym-no-broker-test-group";
+ ConfigResource groupResource = new ConfigResource(ConfigResource.Type.GROUP, group);
+
+ // Verify default config for a config with no broker synonym.
+ // Expected synonym chain: DEFAULT_CONFIG
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_DEFAULT,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set group override; synonyms use group config name since there is no broker synonym.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG, "earliest"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ "earliest",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+ }
+ }
+
+ @ClusterTest(types = {Type.KRAFT},
+ serverProperties = {
+ @ClusterConfigProperty(
+ key = GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, value = "2000")
+ })
+ public void testDescribeGroupConfigSynonymsWithStaticBrokerConfig(ClusterInstance cluster) throws Exception {
+ try (Admin admin = cluster.admin()) {
+ String group = "synonym-static-test-group";
+ ConfigResource groupResource = new ConfigResource(ConfigResource.Type.GROUP, group);
+ ConfigResource brokerResource = new ConfigResource(ConfigResource.Type.BROKER, "0");
+ ConfigResource brokerDefaultResource = new ConfigResource(ConfigResource.Type.BROKER, "");
+
+ // Verify static broker config is reflected in synonyms.
+ // Expected synonym chain: STATIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "2000",
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set group override; it takes precedence over static broker config.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> STATIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG, "3000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "3000",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Add dynamic default broker config.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DYNAMIC_DEFAULT_BROKER_CONFIG
+ // -> STATIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerDefaultResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "4000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "3000",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG)));
+
+ // Add per-broker dynamic config to complete the full 5-layer synonym chain.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DYNAMIC_BROKER_CONFIG -> DYNAMIC_DEFAULT_BROKER_CONFIG
+ // -> STATIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "5000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "3000",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Delete group override; value falls back to per-broker dynamic config.
+ // Expected synonym chain: DYNAMIC_BROKER_CONFIG -> DYNAMIC_DEFAULT_BROKER_CONFIG
+ // -> STATIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG, ""),
+ AlterConfigOp.OpType.DELETE)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "5000",
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG)));
+ }
+ }
+
+ private void assertGroupConfig(
Review Comment:
nit: static?
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/admin/DynamicGroupConfigIntegrationTest.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.test.ClusterInstance;
+import org.apache.kafka.common.test.api.ClusterConfigProperty;
+import org.apache.kafka.common.test.api.ClusterTest;
+import org.apache.kafka.common.test.api.Type;
+import org.apache.kafka.coordinator.group.GroupConfig;
+import org.apache.kafka.coordinator.group.GroupCoordinatorConfig;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class DynamicGroupConfigIntegrationTest {
+
+ @ClusterTest(types = {Type.KRAFT})
+ public void testDescribeGroupConfigSynonymsWithBrokerSynonym(ClusterInstance cluster) throws Exception {
+ try (Admin admin = cluster.admin()) {
+ String group = "synonym-test-group";
+ ConfigResource groupResource = new ConfigResource(ConfigResource.Type.GROUP, group);
+ ConfigResource brokerResource = new ConfigResource(ConfigResource.Type.BROKER, "0");
+ ConfigResource brokerDefaultResource = new ConfigResource(ConfigResource.Type.BROKER, "");
+
+ // Verify default config only.
+ // Expected synonym chain: DEFAULT_CONFIG
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ String.valueOf(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_DEFAULT),
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set per-broker dynamic config.
+ // Expected synonym chain: DYNAMIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "1500"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "1500",
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set dynamic default broker config; per-broker config still takes precedence.
+ // Expected synonym chain: DYNAMIC_BROKER_CONFIG -> DYNAMIC_DEFAULT_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerDefaultResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "2000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "1500",
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set group override; it takes precedence over all broker configs.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DYNAMIC_BROKER_CONFIG
+ // -> DYNAMIC_DEFAULT_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG, "3000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "3000",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+ }
+ }
+
+ @ClusterTest(types = {Type.KRAFT})
+ public void testDescribeGroupConfigSynonymsWithoutBrokerSynonym(ClusterInstance cluster) throws Exception {
+ try (Admin admin = cluster.admin()) {
+ String group = "synonym-no-broker-test-group";
+ ConfigResource groupResource = new ConfigResource(ConfigResource.Type.GROUP, group);
+
+ // Verify default config for a config with no broker synonym.
+ // Expected synonym chain: DEFAULT_CONFIG
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_DEFAULT,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set group override; synonyms use group config name since there is no broker synonym.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG, "earliest"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ "earliest",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+ }
+ }
+
+ @ClusterTest(types = {Type.KRAFT},
+ serverProperties = {
+ @ClusterConfigProperty(
+ key = GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, value = "2000")
+ })
+ public void testDescribeGroupConfigSynonymsWithStaticBrokerConfig(ClusterInstance cluster) throws Exception {
+ try (Admin admin = cluster.admin()) {
+ String group = "synonym-static-test-group";
+ ConfigResource groupResource = new ConfigResource(ConfigResource.Type.GROUP, group);
+ ConfigResource brokerResource = new ConfigResource(ConfigResource.Type.BROKER, "0");
+ ConfigResource brokerDefaultResource = new ConfigResource(ConfigResource.Type.BROKER, "");
+
+ // Verify static broker config is reflected in synonyms.
+ // Expected synonym chain: STATIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "2000",
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set group override; it takes precedence over static broker config.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> STATIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG, "3000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "3000",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Add dynamic default broker config.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DYNAMIC_DEFAULT_BROKER_CONFIG
+ // -> STATIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerDefaultResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "4000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "3000",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG)));
+
+ // Add per-broker dynamic config to complete the full 5-layer synonym chain.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DYNAMIC_BROKER_CONFIG -> DYNAMIC_DEFAULT_BROKER_CONFIG
+ // -> STATIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "5000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "3000",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Delete group override; value falls back to per-broker dynamic config.
+ // Expected synonym chain: DYNAMIC_BROKER_CONFIG -> DYNAMIC_DEFAULT_BROKER_CONFIG
+ // -> STATIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG, ""),
+ AlterConfigOp.OpType.DELETE)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "5000",
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG)));
+ }
+ }
+
+ private void assertGroupConfig(
+ Admin admin,
+ ConfigResource groupResource,
+ String configKey,
+ String expectedValue,
+ ConfigEntry.ConfigSource expectedSource,
+ List<Map.Entry<String, ConfigEntry.ConfigSource>> expectedSynonyms
+ ) throws Exception {
+ DescribeConfigsOptions options = new DescribeConfigsOptions().includeSynonyms(true);
+ ConfigEntry entry = admin.describeConfigs(List.of(groupResource), options)
+ .all().get().get(groupResource).get(configKey);
+ assertEquals(expectedValue, entry.value());
+ assertEquals(expectedSource, entry.source());
+ assertEquals(expectedSynonyms, entry.synonyms().stream()
+ .map(s -> Map.entry(s.name(), s.source()))
+ .toList());
Review Comment:
nit: Should we rather use a Set to unsure that we don't have ordering issues? Or is the order always guaranteed?
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/admin/DynamicGroupConfigIntegrationTest.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.test.ClusterInstance;
+import org.apache.kafka.common.test.api.ClusterConfigProperty;
+import org.apache.kafka.common.test.api.ClusterTest;
+import org.apache.kafka.common.test.api.Type;
+import org.apache.kafka.coordinator.group.GroupConfig;
+import org.apache.kafka.coordinator.group.GroupCoordinatorConfig;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class DynamicGroupConfigIntegrationTest {
+
+ @ClusterTest(types = {Type.KRAFT})
+ public void testDescribeGroupConfigSynonymsWithBrokerSynonym(ClusterInstance cluster) throws Exception {
+ try (Admin admin = cluster.admin()) {
+ String group = "synonym-test-group";
+ ConfigResource groupResource = new ConfigResource(ConfigResource.Type.GROUP, group);
+ ConfigResource brokerResource = new ConfigResource(ConfigResource.Type.BROKER, "0");
+ ConfigResource brokerDefaultResource = new ConfigResource(ConfigResource.Type.BROKER, "");
+
+ // Verify default config only.
+ // Expected synonym chain: DEFAULT_CONFIG
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ String.valueOf(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_DEFAULT),
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set per-broker dynamic config.
+ // Expected synonym chain: DYNAMIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "1500"),
+ AlterConfigOp.OpType.SET)
Review Comment:
I find it a little easier to read like this but I leave it up to you.
```suggestion
new AlterConfigOp(
new ConfigEntry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "1500"),
AlterConfigOp.OpType.SET
)
```
There are other cases.
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/admin/DynamicGroupConfigIntegrationTest.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.test.ClusterInstance;
+import org.apache.kafka.common.test.api.ClusterConfigProperty;
+import org.apache.kafka.common.test.api.ClusterTest;
+import org.apache.kafka.common.test.api.Type;
+import org.apache.kafka.coordinator.group.GroupConfig;
+import org.apache.kafka.coordinator.group.GroupCoordinatorConfig;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class DynamicGroupConfigIntegrationTest {
+
+ @ClusterTest(types = {Type.KRAFT})
+ public void testDescribeGroupConfigSynonymsWithBrokerSynonym(ClusterInstance cluster) throws Exception {
+ try (Admin admin = cluster.admin()) {
+ String group = "synonym-test-group";
+ ConfigResource groupResource = new ConfigResource(ConfigResource.Type.GROUP, group);
+ ConfigResource brokerResource = new ConfigResource(ConfigResource.Type.BROKER, "0");
+ ConfigResource brokerDefaultResource = new ConfigResource(ConfigResource.Type.BROKER, "");
+
+ // Verify default config only.
+ // Expected synonym chain: DEFAULT_CONFIG
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ String.valueOf(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_DEFAULT),
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set per-broker dynamic config.
+ // Expected synonym chain: DYNAMIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "1500"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "1500",
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set dynamic default broker config; per-broker config still takes precedence.
+ // Expected synonym chain: DYNAMIC_BROKER_CONFIG -> DYNAMIC_DEFAULT_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerDefaultResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "2000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "1500",
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set group override; it takes precedence over all broker configs.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DYNAMIC_BROKER_CONFIG
+ // -> DYNAMIC_DEFAULT_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG, "3000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "3000",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+ }
+ }
+
+ @ClusterTest(types = {Type.KRAFT})
+ public void testDescribeGroupConfigSynonymsWithoutBrokerSynonym(ClusterInstance cluster) throws Exception {
+ try (Admin admin = cluster.admin()) {
+ String group = "synonym-no-broker-test-group";
+ ConfigResource groupResource = new ConfigResource(ConfigResource.Type.GROUP, group);
+
+ // Verify default config for a config with no broker synonym.
+ // Expected synonym chain: DEFAULT_CONFIG
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_DEFAULT,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set group override; synonyms use group config name since there is no broker synonym.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG, "earliest"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ "earliest",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+ }
+ }
+
+ @ClusterTest(types = {Type.KRAFT},
+ serverProperties = {
+ @ClusterConfigProperty(
+ key = GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, value = "2000")
Review Comment:
nit: We usually put the key and the value on the same line as `@ClusterConfigProperty`.
##########
core/src/main/scala/kafka/server/ConfigHelper.scala:
##########
@@ -161,15 +161,23 @@ class ConfigHelper(metadataCache: MetadataCache, config: KafkaConfig, configRepo
private def createGroupConfigEntry(groupConfig: GroupConfig, groupProps: Properties, includeSynonyms: Boolean, includeDocumentation: Boolean)
(name: String, value: Any): DescribeConfigsResponseData.DescribeConfigsResourceResult = {
- val allNames = brokerSynonyms(name)
val configEntryType = GroupConfig.configType(name).toScala
val isSensitive = KafkaConfig.maybeSensitive(configEntryType)
val valueAsString = if (isSensitive) null else ConfigDef.convertToString(value, configEntryType.orNull)
val allSynonyms = {
- val list = configSynonyms(name, allNames, isSensitive)
+ val brokerConfigName = GroupConfig.ALL_GROUP_CONFIG_SYNONYMS.get(name)
Review Comment:
nit: Should we introduce an helper method in `GroupConfig` like `synonym` which always return an Optional? It would simplify the handling below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r2989654531
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorConfig.java:
##########
@@ -392,6 +392,8 @@ public class GroupCoordinatorConfig {
STREAMS_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
STREAMS_GROUP_ASSIGNOR_OFFLOAD_ENABLE_CONFIG
);
+
+ public static final String GROUP_PREFIX = "group.";
Review Comment:
Thanks for the suggestion. I've added `testGroupConfigBrokerSynonymNamingConvention` in `GroupCoordinatorConfigTest`.
It iterates all `GroupConfig` entries and verifies each has a corresponding broker config named `GROUP_PREFIX + groupConfigName`. Entries without broker-level synonyms are maintained in an explicit exclusion list, and a reverse check ensures the exclusion list stays accurate.
Any deviation from the naming convention will fail CI with a clear message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r2981859112
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorConfig.java:
##########
@@ -392,6 +392,8 @@ public class GroupCoordinatorConfig {
STREAMS_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
STREAMS_GROUP_ASSIGNOR_OFFLOAD_ENABLE_CONFIG
);
+
+ public static final String GROUP_PREFIX = "group.";
Review Comment:
Thanks for the review. Both approaches are valid.
I chose the prefix approach because Group configs have a uniform naming convention — every broker config name is "group." + name, with no exceptions. Given the current convention, future group configs would likely follow the same naming rule as well, so new configs wouldn't require maintaining a separate mapping table.
Topic needs `ALL_TOPIC_CONFIG_SYNONYMS` because its naming is irregular — different prefixes (`log., log.cleaner.`), different names (`flush.messages → log.flush.interval.messages`), and unit conversions. Group configs don't have this complexity.
I'm also open to introducing `ALL_GROUP_CONFIG_SYNONYMS` if we think the explicitness of a mapping table outweighs the cost of maintaining it for each new group config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3022726858
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/admin/DynamicGroupConfigIntegrationTest.java:
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.test.ClusterInstance;
+import org.apache.kafka.common.test.api.ClusterConfigProperty;
+import org.apache.kafka.common.test.api.ClusterTest;
+import org.apache.kafka.common.test.api.Type;
+import org.apache.kafka.coordinator.group.GroupConfig;
+import org.apache.kafka.coordinator.group.GroupCoordinatorConfig;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class DynamicGroupConfigIntegrationTest {
+
+ @ClusterTest(types = {Type.KRAFT})
+ public void testDescribeGroupConfigSynonymsWithBrokerSynonym(ClusterInstance cluster) throws Exception {
+ try (Admin admin = cluster.admin()) {
+ String group = "synonym-test-group";
+ ConfigResource groupResource = new ConfigResource(ConfigResource.Type.GROUP, group);
+ ConfigResource brokerResource = new ConfigResource(ConfigResource.Type.BROKER, "0");
+ ConfigResource brokerDefaultResource = new ConfigResource(ConfigResource.Type.BROKER, "");
+
+ // Verify default config only.
+ // Expected synonym chain: DEFAULT_CONFIG
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ String.valueOf(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_DEFAULT),
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set per-broker dynamic config.
+ // Expected synonym chain: DYNAMIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "1500"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "1500",
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set dynamic default broker config; per-broker config still takes precedence.
+ // Expected synonym chain: DYNAMIC_BROKER_CONFIG -> DYNAMIC_DEFAULT_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerDefaultResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "2000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "1500",
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set group override; it takes precedence over all broker configs.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DYNAMIC_BROKER_CONFIG
+ // -> DYNAMIC_DEFAULT_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG, "3000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "3000",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+ }
+ }
+
+ @ClusterTest(types = {Type.KRAFT})
+ public void testDescribeGroupConfigSynonymsWithoutBrokerSynonym(ClusterInstance cluster) throws Exception {
+ try (Admin admin = cluster.admin()) {
+ String group = "synonym-no-broker-test-group";
+ ConfigResource groupResource = new ConfigResource(ConfigResource.Type.GROUP, group);
+
+ // Verify default config for a config with no broker synonym.
+ // Expected synonym chain: DEFAULT_CONFIG
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_DEFAULT,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set group override; synonyms use group config name since there is no broker synonym.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG, "earliest"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ "earliest",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+ }
+ }
+
+ @ClusterTest(types = {Type.KRAFT},
+ serverProperties = {
+ @ClusterConfigProperty(
+ key = GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, value = "2000")
+ })
+ public void testDescribeGroupConfigSynonymsWithStaticBrokerConfig(ClusterInstance cluster) throws Exception {
+ try (Admin admin = cluster.admin()) {
+ String group = "synonym-static-test-group";
+ ConfigResource groupResource = new ConfigResource(ConfigResource.Type.GROUP, group);
+ ConfigResource brokerResource = new ConfigResource(ConfigResource.Type.BROKER, "0");
+ ConfigResource brokerDefaultResource = new ConfigResource(ConfigResource.Type.BROKER, "");
+
+ // Verify static broker config is reflected in synonyms.
+ // Expected synonym chain: STATIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "2000",
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Set group override; it takes precedence over static broker config.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> STATIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG, "3000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "3000",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Add dynamic default broker config.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DYNAMIC_DEFAULT_BROKER_CONFIG
+ // -> STATIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerDefaultResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "4000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "3000",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG)));
+
+ // Add per-broker dynamic config to complete the full 5-layer synonym chain.
+ // Expected synonym chain: DYNAMIC_GROUP_CONFIG -> DYNAMIC_BROKER_CONFIG -> DYNAMIC_DEFAULT_BROKER_CONFIG
+ // -> STATIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(brokerResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG, "5000"),
+ AlterConfigOp.OpType.SET)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "3000",
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG,
+ List.of(
+ Map.entry(GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_GROUP_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG))
+ );
+
+ // Delete group override; value falls back to per-broker dynamic config.
+ // Expected synonym chain: DYNAMIC_BROKER_CONFIG -> DYNAMIC_DEFAULT_BROKER_CONFIG
+ // -> STATIC_BROKER_CONFIG -> DEFAULT_CONFIG
+ admin.incrementalAlterConfigs(Map.of(groupResource, List.of(
+ new AlterConfigOp(new ConfigEntry(
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG, ""),
+ AlterConfigOp.OpType.DELETE)
+ ))).all().get();
+ cluster.ensureConsistentMetadata();
+
+ assertGroupConfig(
+ admin,
+ groupResource,
+ GroupConfig.CONSUMER_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ "5000",
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG,
+ List.of(
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DYNAMIC_DEFAULT_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.STATIC_BROKER_CONFIG),
+ Map.entry(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
+ ConfigEntry.ConfigSource.DEFAULT_CONFIG)));
+ }
+ }
+
+ private void assertGroupConfig(
+ Admin admin,
+ ConfigResource groupResource,
+ String configKey,
+ String expectedValue,
+ ConfigEntry.ConfigSource expectedSource,
+ List<Map.Entry<String, ConfigEntry.ConfigSource>> expectedSynonyms
+ ) throws Exception {
+ DescribeConfigsOptions options = new DescribeConfigsOptions().includeSynonyms(true);
+ ConfigEntry entry = admin.describeConfigs(List.of(groupResource), options)
+ .all().get().get(groupResource).get(configKey);
+ assertEquals(expectedValue, entry.value());
+ assertEquals(expectedSource, entry.source());
+ assertEquals(expectedSynonyms, entry.synonyms().stream()
+ .map(s -> Map.entry(s.name(), s.source()))
+ .toList());
Review Comment:
The order is guaranteed, synonyms are listed by priority (highest to lowest). Verifying the order is part of what these tests validate, so using a List is preferred here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r2982964112
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorConfig.java:
##########
@@ -392,6 +392,8 @@ public class GroupCoordinatorConfig {
STREAMS_GROUP_ASSIGNMENT_INTERVAL_MS_CONFIG,
STREAMS_GROUP_ASSIGNOR_OFFLOAD_ENABLE_CONFIG
);
+
+ public static final String GROUP_PREFIX = "group.";
Review Comment:
> I chose the prefix approach because Group configs have a uniform naming convention — every broker config name is "group." + name, with no exceptions. Given the current convention, future group configs would likely follow the same naming rule as well, so new configs wouldn't require maintaining a separate mapping table.
My primary reservation is that predicating this logic on the `group.` prefix introduces a tacit constraint that is highly susceptible to oversight by future developers. For example, in https://github.com/apache/kafka/pull/21801#discussion_r2952177369, there was an attempt to add "group.streams.max.num.warmup.replicas" and "streams.max.warmup.replicas", which directly contravenes the exact mapping convention assumed in this PR.
I'm amenable to adopting the `group.` prefix approach, but would you mind adding a unit test to codify this consistency? we could leverage reflection to cross-reference all configs between GroupConfig and GroupCoordinatorConfig, coorborating that the mapping always holds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3001988063
##########
core/src/main/scala/kafka/server/ConfigHelper.scala:
##########
@@ -161,15 +161,22 @@ class ConfigHelper(metadataCache: MetadataCache, config: KafkaConfig, configRepo
private def createGroupConfigEntry(groupConfig: GroupConfig, groupProps: Properties, includeSynonyms: Boolean, includeDocumentation: Boolean)
(name: String, value: Any): DescribeConfigsResponseData.DescribeConfigsResourceResult = {
- val allNames = brokerSynonyms(name)
val configEntryType = GroupConfig.configType(name).toScala
val isSensitive = KafkaConfig.maybeSensitive(configEntryType)
val valueAsString = if (isSensitive) null else ConfigDef.convertToString(value, configEntryType.orNull)
val allSynonyms = {
- val list = configSynonyms(name, allNames, isSensitive)
+ val brokerConfigName = GroupCoordinatorConfig.GROUP_PREFIX + name
+ val list = if (KafkaConfig.configDef.configKeys().containsKey(brokerConfigName))
Review Comment:
As for internal configs, `createResponseConfig` already uses `AbstractConfig.nonInternalValues() `to filter them out, so future internal group configs would be excluded automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #21855:
URL: https://github.com/apache/kafka/pull/21855#issuecomment-4142848200
@majialoong Thanks for all the cases! Do we have test cases for each of them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on PR #21855:
URL: https://github.com/apache/kafka/pull/21855#issuecomment-4144403118
> @majialoong Thanks for all the cases! Do we have test cases for each of them?
@dajac The current `testDescribeConfigsGroupSynonyms` covers some scenarios. I'll add more test cases for the remaining ones and split them into separate methods so each test focuses on a single synonym chain pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #21855:
URL: https://github.com/apache/kafka/pull/21855#discussion_r3002684448
##########
core/src/main/scala/kafka/server/ConfigHelper.scala:
##########
@@ -161,15 +161,22 @@ class ConfigHelper(metadataCache: MetadataCache, config: KafkaConfig, configRepo
private def createGroupConfigEntry(groupConfig: GroupConfig, groupProps: Properties, includeSynonyms: Boolean, includeDocumentation: Boolean)
(name: String, value: Any): DescribeConfigsResponseData.DescribeConfigsResourceResult = {
- val allNames = brokerSynonyms(name)
val configEntryType = GroupConfig.configType(name).toScala
val isSensitive = KafkaConfig.maybeSensitive(configEntryType)
val valueAsString = if (isSensitive) null else ConfigDef.convertToString(value, configEntryType.orNull)
val allSynonyms = {
- val list = configSynonyms(name, allNames, isSensitive)
+ val brokerConfigName = GroupCoordinatorConfig.GROUP_PREFIX + name
+ val list = if (KafkaConfig.configDef.configKeys().containsKey(brokerConfigName))
Review Comment:
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org