You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/02/03 16:53:51 UTC
[GitHub] [kafka] rondagostino opened a new pull request #10039: MINOR: Defer log recovery until LogManager startup
rondagostino opened a new pull request #10039:
URL: https://github.com/apache/kafka/pull/10039
Currently log recovery begins as soon as we instantiate `LogManager`, but when using a Raft-based metadata quorum we won't have configs until after we catch up on the metadata log. We therefore defer log recovery until we actually invoke `startup()` on the `LogManager` instance. This timing difference has no effect when using ZooKeeper because we immediately invoke `startup()` on the instantiated instance, but it gives us the necessary flexibility for accurate log recovery with updated configs when using a Raft-based metadata quorum.
We also convert `LogManager` to use a `ConfigRepository` to load topic configs (which can override the default log configs) instead of having a hard-coded dependency on ZooKeeper. We retrieve the topic configs when we invoke `startup()` -- which again is effectively no different from a timing perspective than what we do today for the ZooKeeper case.
This PR depends upon https://github.com/apache/kafka/pull/10005/ (either that PR being merged, or this one should be subsumed into that one).
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#issuecomment-774790663
Merged to trunk and 2.8 branches.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#discussion_r569580157
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -402,7 +403,26 @@ class LogManager(logDirs: Seq[File],
/**
* Start the background threads to flush logs and do log cleanup
*/
- def startup(): Unit = {
+ def startup(retrieveTopicNames: () => Set[String]): Unit = {
+ startup(generateTopicLogConfigs(retrieveTopicNames))
+ }
+
+ // visible for testing
+ private[log] def generateTopicLogConfigs(retrieveTopicNames: () => Set[String]): Map[String, LogConfig] = {
+ val topicLogConfigs: mutable.Map[String, LogConfig] = mutable.Map()
+ val defaultProps = currentDefaultConfig.originals()
+ retrieveTopicNames().foreach { topicName =>
+ val overrides = configRepository.topicConfigs(topicName)
+ val logConfig = LogConfig.fromProps(defaultProps, overrides)
+ topicLogConfigs(topicName) = logConfig
+ }
Review comment:
I wonder about the efficiency of this loop when using ZooKeeper. Today we ask ZooKeeper for all of the topics, then we ask ZooKeeper for all of the configs for all of those topics. However, under the hood it does appear that `KafkaZkClient.getTopicConfigs(Set)` is iterating through all of the topics and querying each one individually. So I raise it just to get another pair of eyes to confirm that this explicit loop will not perform worse than what appears to be a single round-trip but is in fact not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#discussion_r571275654
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -403,7 +401,30 @@ class LogManager(logDirs: Seq[File],
/**
* Start the background threads to flush logs and do log cleanup
*/
- def startup(): Unit = {
+ def startup(retrieveTopicNames: => Set[String]): Unit = {
Review comment:
Would like to keep it as a verb: `retrieveTopicsToLoad`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
rondagostino commented on pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#issuecomment-774777518
Latest changes LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a change in pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
ijuma commented on a change in pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#discussion_r571642087
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -118,8 +116,6 @@ class LogManager(logDirs: Seq[File],
logDirsSet
}
- loadLogs()
Review comment:
Looks like we need to look if it's still safe for the things after this to be run in the constructor. Perhaps the cleaner should be started after the logs are loaded?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#discussion_r569602196
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -402,7 +403,26 @@ class LogManager(logDirs: Seq[File],
/**
* Start the background threads to flush logs and do log cleanup
*/
- def startup(): Unit = {
+ def startup(retrieveTopicNames: () => Set[String]): Unit = {
+ startup(generateTopicLogConfigs(retrieveTopicNames))
+ }
+
+ // visible for testing
+ private[log] def generateTopicLogConfigs(retrieveTopicNames: () => Set[String]): Map[String, LogConfig] = {
+ val topicLogConfigs: mutable.Map[String, LogConfig] = mutable.Map()
+ val defaultProps = currentDefaultConfig.originals()
+ retrieveTopicNames().foreach { topicName =>
+ val overrides = configRepository.topicConfigs(topicName)
+ val logConfig = LogConfig.fromProps(defaultProps, overrides)
+ topicLogConfigs(topicName) = logConfig
Review comment:
only add it if there are some overides
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a change in pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
ijuma commented on a change in pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#discussion_r571672398
##########
File path: core/src/test/scala/unit/kafka/cluster/PartitionLockTest.scala
##########
@@ -305,15 +304,6 @@ class PartitionLockTest extends Logging {
partition
}
- private def createLogProperties(overrides: Map[String, String]): Properties = {
Review comment:
The tests pass without this, but we may want to pass these to the LogManager.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#issuecomment-774790209
The JDK 8 build had unrelated flaky test failures this time while JDK 11 and 15 terminated before completion (again unrelated). Will go ahead and merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#discussion_r571274481
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -403,7 +401,30 @@ class LogManager(logDirs: Seq[File],
/**
* Start the background threads to flush logs and do log cleanup
*/
- def startup(): Unit = {
+ def startup(retrieveTopicNames: => Set[String]): Unit = {
+ startupWithTopicLogConfigOverrides(generateTopicLogConfigs(retrieveTopicNames))
Review comment:
Yes, `startup()` is invoked immediately after the instance is constructed when we are using ZooKeeper, and we can get the configs at that point since we can just query ZK.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#discussion_r569600606
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -256,9 +254,10 @@ class LogManager(logDirs: Seq[File],
private[log] def loadLog(logDir: File,
hadCleanShutdown: Boolean,
recoveryPoints: Map[TopicPartition, Long],
- logStartOffsets: Map[TopicPartition, Long]): Log = {
+ logStartOffsets: Map[TopicPartition, Long],
+ topicLogConfigs: Map[String, LogConfig]): Log = {
val topicPartition = Log.parseTopicPartitionName(logDir)
- val config = topicConfigs.getOrElse(topicPartition.topic, currentDefaultConfig)
+ val config = topicLogConfigs.getOrElse(topicPartition.topic, currentDefaultConfig)
Review comment:
We can ignore any topic that has no explicit configs in `startup()` below since this is doing `.getOrElse()` -- it'll save memory.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#issuecomment-774775433
One unrelated flaky failure:
`Build / JDK 15 / kafka.controller.ControllerIntegrationTest.testPartitionReassignmentToBrokerWithOfflineLogDir()`
One of the builds exited early, which is unrelated too. To be sure, restarted the tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#discussion_r571278884
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -352,13 +348,15 @@ class LogManager(logDirs: Seq[File],
val numLogsLoaded = new AtomicInteger(0)
numTotalLogs += logsToLoad.length
- val jobsForDir = logsToLoad.map { logDir =>
+ val jobsForDir = logsToLoad
+ .filter(logDir => Log.parseTopicPartitionName(logDir).topic != KafkaRaftServer.MetadataTopic)
Review comment:
Yeah, we use the list of topics just to know what to check for log config overrides. This will see anything in the log dir. Now that I realize it, we could probably use this to know what topics we need to look at for log config overrides. But I don't want to change too much at this point.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mumrah commented on a change in pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
mumrah commented on a change in pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#discussion_r571268386
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -403,7 +401,30 @@ class LogManager(logDirs: Seq[File],
/**
* Start the background threads to flush logs and do log cleanup
*/
- def startup(): Unit = {
+ def startup(retrieveTopicNames: => Set[String]): Unit = {
+ startupWithTopicLogConfigOverrides(generateTopicLogConfigs(retrieveTopicNames))
Review comment:
We are able to load the topic configs right away because they are coming from ZK, right?
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -403,7 +401,30 @@ class LogManager(logDirs: Seq[File],
/**
* Start the background threads to flush logs and do log cleanup
*/
- def startup(): Unit = {
+ def startup(retrieveTopicNames: => Set[String]): Unit = {
+ startupWithTopicLogConfigOverrides(generateTopicLogConfigs(retrieveTopicNames))
+ }
+
+ // visible for testing
+ private[log] def generateTopicLogConfigs(topicNames: Set[String]): Map[String, LogConfig] = {
+ val topicLogConfigs: mutable.Map[String, LogConfig] = mutable.Map()
Review comment:
nit: can move type info to right hand side and just have `val topicLogConfigs = ...`
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -403,7 +401,30 @@ class LogManager(logDirs: Seq[File],
/**
* Start the background threads to flush logs and do log cleanup
*/
- def startup(): Unit = {
+ def startup(retrieveTopicNames: => Set[String]): Unit = {
Review comment:
Feels slightly odd to pass in the set of topics to load here, but I can't think of a good way to avoid it. Perhaps we could pass MetadataCache into LogManager and let startup call MetadataCache#getAllTopics? That might be more risky though since it changes the startup order in KafkaServer, maybe we can look into this as a follow-up.
Besides that, the name here seems strange. Maybe something like "topicsToLoad"?
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -352,13 +348,15 @@ class LogManager(logDirs: Seq[File],
val numLogsLoaded = new AtomicInteger(0)
numTotalLogs += logsToLoad.length
- val jobsForDir = logsToLoad.map { logDir =>
+ val jobsForDir = logsToLoad
+ .filter(logDir => Log.parseTopicPartitionName(logDir).topic != KafkaRaftServer.MetadataTopic)
Review comment:
Will the metadata topic get passed into LogManager? I would guess not since it's a Raft topic and not a regular Kafka topic.
Also style nit: you can do `logsToLoad.filter { logDir => ... }` similar to the `map` below
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#discussion_r571276941
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -403,7 +401,30 @@ class LogManager(logDirs: Seq[File],
/**
* Start the background threads to flush logs and do log cleanup
*/
- def startup(): Unit = {
+ def startup(retrieveTopicNames: => Set[String]): Unit = {
Review comment:
Actually, this isn't the list of topics to load -- it's the list of topics for which we will look for log config overrides. `retrieveTopicsForLogConfigOverrideCheck`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#discussion_r571470926
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -352,13 +348,15 @@ class LogManager(logDirs: Seq[File],
val numLogsLoaded = new AtomicInteger(0)
numTotalLogs += logsToLoad.length
- val jobsForDir = logsToLoad.map { logDir =>
+ val jobsForDir = logsToLoad
+ .filter(logDir => Log.parseTopicPartitionName(logDir).topic != KafkaRaftServer.MetadataTopic)
Review comment:
I suspect we shouldn’t load topic configs on-demand because it would change the current behavior. Specifically, if we can’t load the configs, currently the object instantiation will raise an exception. The current patch will cause the exception to be raised in startup(), which is effectively the same thing. But if we load them on-demand then both instantiation and startup() will succeed, and the exception will occur in a thread doing recovery.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma merged pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
ijuma merged pull request #10039:
URL: https://github.com/apache/kafka/pull/10039
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#discussion_r571283864
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -352,13 +348,15 @@ class LogManager(logDirs: Seq[File],
val numLogsLoaded = new AtomicInteger(0)
numTotalLogs += logsToLoad.length
- val jobsForDir = logsToLoad.map { logDir =>
+ val jobsForDir = logsToLoad
+ .filter(logDir => Log.parseTopicPartitionName(logDir).topic != KafkaRaftServer.MetadataTopic)
Review comment:
We could get rid of the parameter in `startup()` and create a 'TopicLogConfigOverrideRetriever` class that held the `ConfigRepository` and would retrieve any topic configs in a thread-safe manner -- then we just pass that to `loadLog()` and it asks for the overrides, and the class would retrieve iff they haven't been retrieved before. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#discussion_r571283864
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -352,13 +348,15 @@ class LogManager(logDirs: Seq[File],
val numLogsLoaded = new AtomicInteger(0)
numTotalLogs += logsToLoad.length
- val jobsForDir = logsToLoad.map { logDir =>
+ val jobsForDir = logsToLoad
+ .filter(logDir => Log.parseTopicPartitionName(logDir).topic != KafkaRaftServer.MetadataTopic)
Review comment:
We could get rid of the parameter in `startup()` and create a `TopicLogConfigOverrideRetriever` class that held the `ConfigRepository` and would retrieve any topic configs in a thread-safe manner -- then we just pass that to `loadLog()` and it asks for the overrides, and the class would retrieve iff they haven't been retrieved before. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10039: MINOR: Defer log recovery until LogManager startup
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10039:
URL: https://github.com/apache/kafka/pull/10039#discussion_r571642788
##########
File path: core/src/main/scala/kafka/log/LogManager.scala
##########
@@ -118,8 +116,6 @@ class LogManager(logDirs: Seq[File],
logDirsSet
}
- loadLogs()
Review comment:
Good point. Yes, it seems safer to delay starting anything until we actually have recovered.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org