You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/12/06 01:35:21 UTC
[GitHub] [kafka] cmccabe opened a new pull request, #12958: MINOR: Move dynamic config logic to DynamicConfigPublisher
cmccabe opened a new pull request, #12958:
URL: https://github.com/apache/kafka/pull/12958
Split out the logic for applying dynamic configurations to a KafkaConfig object from BrokerMetadataPublisher into a new class, DynamicConfigPublisher. This will allow the ControllerServer to also run this code, in a follow-up change.
Create separate KafkaConfig objects in BrokerServer versus ControllerServer. This is necessary because the controller will apply configuration changes as soon as its raft client catches up to the high water mark, whereas the broker will wait for the active controller to acknowledge it has caught up in a heartbeat response. So when running in combined mode, we want two separate KafkaConfig objects that are changed at different times.
Minor changes: improve the error message when catching up broker metadata fails. Fix incorrect indentation in checkstyle/import-control.xml. Invoke AppInfoParser.unregisterAppInfo from SharedServer.stop so that it happens only when both the controller and broker have shut down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe merged pull request #12958: MINOR: Move dynamic config logic to DynamicConfigPublisher
Posted by GitBox <gi...@apache.org>.
cmccabe merged PR #12958:
URL: https://github.com/apache/kafka/pull/12958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mumrah commented on a diff in pull request #12958: MINOR: Move dynamic config logic to DynamicConfigPublisher
Posted by GitBox <gi...@apache.org>.
mumrah commented on code in PR #12958:
URL: https://github.com/apache/kafka/pull/12958#discussion_r1041165700
##########
core/src/main/scala/kafka/server/BrokerServer.scala:
##########
@@ -71,7 +71,7 @@ class BrokerServer(
val initialOfflineDirs: Seq[String],
) extends KafkaBroker {
val threadNamePrefix = sharedServer.threadNamePrefix
- val config = sharedServer.config
+ val config = new KafkaConfig(sharedServer.sharedServerConfig.props, false, None)
Review Comment:
Could we add a method on SharedServer like `createKafkaConfig` rather than expose `sharedServerConfig` to BrokerServer/ControllerServer? I worry someone down the line might read from the shared config rather than the local one thats getting dynamic updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #12958: MINOR: Move dynamic config logic to DynamicConfigPublisher
Posted by GitBox <gi...@apache.org>.
cmccabe commented on code in PR #12958:
URL: https://github.com/apache/kafka/pull/12958#discussion_r1041503192
##########
core/src/main/scala/kafka/server/BrokerServer.scala:
##########
@@ -71,7 +71,7 @@ class BrokerServer(
val initialOfflineDirs: Seq[String],
) extends KafkaBroker {
val threadNamePrefix = sharedServer.threadNamePrefix
- val config = sharedServer.config
+ val config = new KafkaConfig(sharedServer.sharedServerConfig.props, false, None)
Review Comment:
Good idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on pull request #12958: MINOR: Move dynamic config logic to DynamicConfigPublisher
Posted by GitBox <gi...@apache.org>.
cmccabe commented on PR #12958:
URL: https://github.com/apache/kafka/pull/12958#issuecomment-1341414560
failing test is unrelated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mumrah commented on a diff in pull request #12958: MINOR: Move dynamic config logic to DynamicConfigPublisher
Posted by GitBox <gi...@apache.org>.
mumrah commented on code in PR #12958:
URL: https://github.com/apache/kafka/pull/12958#discussion_r1041510271
##########
core/src/main/scala/kafka/server/metadata/DynamicConfigPublisher.scala:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.server.metadata
+
+import java.util.Properties
+import kafka.server.ConfigAdminManager.toLoggableProps
+import kafka.server.{ConfigEntityName, ConfigHandler, ConfigType, KafkaConfig}
+import kafka.utils.Logging
+import org.apache.kafka.common.config.ConfigResource.Type.{BROKER, TOPIC}
+import org.apache.kafka.image.{MetadataDelta, MetadataImage}
+import org.apache.kafka.server.fault.FaultHandler
+
+
+class DynamicConfigPublisher(
+ conf: KafkaConfig,
+ faultHandler: FaultHandler,
+ dynamicConfigHandlers: Map[String, ConfigHandler],
+ nodeType: String
+) extends Logging {
+ logIdent = s"[DynamicConfigPublisher nodeType=${nodeType} id=${conf.nodeId}] "
+
+ def publish(delta: MetadataDelta, newImage: MetadataImage): Unit = {
+ val deltaName = s"MetadataDelta up to ${newImage.highestOffsetAndEpoch().offset}"
+ try {
+ // Apply configuration deltas.
+ Option(delta.configsDelta()).foreach { configsDelta =>
+ configsDelta.changes().keySet().forEach { resource =>
+ val props = newImage.configs().configProperties(resource)
+ resource.`type`() match {
+ case TOPIC =>
+ dynamicConfigHandlers.get(ConfigType.Topic).foreach(topicConfigHandler =>
+ try {
+ // Apply changes to a topic's dynamic configuration.
+ info(s"Updating topic ${resource.name()} with new configuration : " +
+ toLoggableProps(resource, props).mkString(","))
+ topicConfigHandler.processConfigChanges(resource.name(), props)
+ } catch {
+ case t: Throwable => faultHandler.handleFault("Error updating topic " +
+ s"${resource.name()} with new configuration: ${toLoggableProps(resource, props).mkString(",")} " +
+ s"in ${deltaName}", t)
+ }
+ )
+ case BROKER =>
Review Comment:
Unrelated to this change, but do we support BROKER_LOGGER configs in KRaft?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org