You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "ijuma (via GitHub)" <gi...@apache.org> on 2024/12/26 16:12:54 UTC
[PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
ijuma opened a new pull request, #18321:
URL: https://github.com/apache/kafka/pull/18321
Convert v0/v1 record batches to v2 during compaction even if said record batches would be
written with no change otherwise.
Test cases we want (some of them may already exist, to be verified):
1. V0 compressed record batch with multiple records is converted into single V2 record batch
2. V0 uncompressed records are converted into single record V2 record batches
3. V0 records are converted to V2 records with timestampType set to the topic config (check both values)
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1904791916
##########
core/src/test/scala/unit/kafka/log/LogCleanerParameterizedIntegrationTest.scala:
##########
@@ -134,6 +135,131 @@ class LogCleanerParameterizedIntegrationTest extends AbstractLogCleanerIntegrati
assertEquals(toMap(messages), toMap(read), "Contents of the map shouldn't change")
}
+ @ParameterizedTest
+ @ArgumentsSource(classOf[LogCleanerParameterizedIntegrationTest.ExcludeZstd])
+ def testCleanerWithMessageFormatV0V1V2(compressionType: CompressionType): Unit = {
+ val compression = Compression.of(compressionType).build()
+ val largeMessageKey = 20
+ val (largeMessageValue, largeMessageSet) = createLargeSingleMessageSet(largeMessageKey, RecordBatch.MAGIC_VALUE_V0, compression)
+ val maxMessageSize = compression match {
+ case Compression.NONE => largeMessageSet.sizeInBytes
+ case _ =>
+ // the broker assigns absolute offsets for message format 0 which potentially causes the compressed size to
+ // increase because the broker offsets are larger than the ones assigned by the client
+ // adding `6` to the message set size is good enough for this test: it covers the increased message size while
+ // still being less than the overhead introduced by the conversion from message format version 0 to 1
+ largeMessageSet.sizeInBytes + 6
Review Comment:
This is probably unnecessary. LogCleaner can grow the readBuffer to larger than maxMessageSize, based on the batch size in the header.
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -509,8 +509,8 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
private def initializeLeaderEpochCache(): Unit = lock synchronized {
- leaderEpochCache = UnifiedLog.maybeCreateLeaderEpochCache(
- dir, topicPartition, logDirFailureChannel, logIdent, leaderEpochCache, scheduler)
+ leaderEpochCache = Some(UnifiedLog.createLeaderEpochCache(
Review Comment:
Ideally, we want to change `leaderEpochCache` to not an `Option`. If it's too much in this PR, could we file a followup jira?
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -757,7 +769,8 @@ class UnifiedLog(@volatile var logStartOffset: Long,
leaderEpoch: Int,
requestLocal: Option[RequestLocal],
verificationGuard: VerificationGuard,
- ignoreRecordSize: Boolean): LogAppendInfo = {
+ ignoreRecordSize: Boolean,
+ toMagic: Byte = RecordBatch.CURRENT_MAGIC_VALUE): LogAppendInfo = {
Review Comment:
For testing, we could append records with `validateAndAssignOffsets=false` and `origin=AppendOrigin.REPLICATION`. This way, we can get rid of `toMagic` here and the `toMagic` in `LogValidator` since it's always 2. This also probably allows us to remove some code in `LogValidator` like `convertAndAssignOffsetsNonCompressed()`. The slight inconvenience is that the caller needs to assign the right offset in records.
##########
core/src/test/scala/unit/kafka/log/LogCleanerParameterizedIntegrationTest.scala:
##########
@@ -134,6 +135,131 @@ class LogCleanerParameterizedIntegrationTest extends AbstractLogCleanerIntegrati
assertEquals(toMap(messages), toMap(read), "Contents of the map shouldn't change")
}
+ @ParameterizedTest
+ @ArgumentsSource(classOf[LogCleanerParameterizedIntegrationTest.ExcludeZstd])
+ def testCleanerWithMessageFormatV0V1V2(compressionType: CompressionType): Unit = {
+ val compression = Compression.of(compressionType).build()
+ val largeMessageKey = 20
+ val (largeMessageValue, largeMessageSet) = createLargeSingleMessageSet(largeMessageKey, RecordBatch.MAGIC_VALUE_V0, compression)
+ val maxMessageSize = compression match {
+ case Compression.NONE => largeMessageSet.sizeInBytes
+ case _ =>
+ // the broker assigns absolute offsets for message format 0 which potentially causes the compressed size to
+ // increase because the broker offsets are larger than the ones assigned by the client
+ // adding `6` to the message set size is good enough for this test: it covers the increased message size while
+ // still being less than the overhead introduced by the conversion from message format version 0 to 1
+ largeMessageSet.sizeInBytes + 6
+ }
+
+ cleaner = makeCleaner(partitions = topicPartitions, maxMessageSize = maxMessageSize)
+
+ val log = cleaner.logs.get(topicPartitions(0))
+ val props = logConfigProperties(maxMessageSize = maxMessageSize)
+ props.put(TopicConfig.MESSAGE_TIMESTAMP_TYPE_CONFIG, TimestampType.LOG_APPEND_TIME.name)
+ val logConfig = new LogConfig(props)
+ log.updateConfig(logConfig)
+
+ val appends1 = writeDups(numKeys = 100, numDups = 3, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V0)
+ val startSize = log.size
+ cleaner.startup()
+
+ val firstDirty = log.activeSegment.baseOffset
+ checkLastCleaned("log", 0, firstDirty)
+ val compactedSize = log.logSegments.asScala.map(_.size).sum
+ assertTrue(startSize > compactedSize, s"log should have been compacted: startSize=$startSize compactedSize=$compactedSize")
+
+ checkLogAfterAppendingDups(log, startSize, appends1)
+
+ val dupsV0 = writeDups(numKeys = 40, numDups = 3, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V0)
+ val appendInfo = log.appendAsLeaderWithRecordVersion(largeMessageSet, leaderEpoch = 0, recordVersion = RecordVersion.V0)
+ // move LSO forward to increase compaction bound
+ log.updateHighWatermark(log.logEndOffset)
+ val largeMessageOffset = appendInfo.firstOffset
+
+ // also add some messages with version 1 and version 2 to check that we handle mixed format versions correctly
+ val dupsV1 = writeDups(startKey = 30, numKeys = 40, numDups = 3, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V1)
+ val dupsV2 = writeDups(startKey = 15, numKeys = 5, numDups = 3, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V2)
+
+ val v0RecordKeysWithNoV1V2Updates = (appends1.map(_._1).toSet -- dupsV1.map(_._1) -- dupsV2.map(_._1)).map(_.toString)
+ val appends2: Seq[(Int, String, Long)] =
+ appends1 ++ dupsV0 ++ Seq((largeMessageKey, largeMessageValue, largeMessageOffset)) ++ dupsV1 ++ dupsV2
+
+ // roll the log so that all appended messages can be compacted
+ log.roll()
+ val firstDirty2 = log.activeSegment.baseOffset
+ checkLastCleaned("log", 0, firstDirty2)
+
+ checkLogAfterAppendingDups(log, startSize, appends2)
+ checkLogAfterConvertingToV2(compressionType, log, logConfig.messageTimestampType, v0RecordKeysWithNoV1V2Updates)
+ }
+
+ @ParameterizedTest
+ @ArgumentsSource(classOf[LogCleanerParameterizedIntegrationTest.ExcludeZstd])
+ def testCleaningNestedMessagesWithV0V1(compressionType: CompressionType): Unit = {
+ val compression = Compression.of(compressionType).build()
+ val maxMessageSize = 192
+ cleaner = makeCleaner(partitions = topicPartitions, maxMessageSize = maxMessageSize, segmentSize = 256)
+
+ val log = cleaner.logs.get(topicPartitions(0))
+ val logConfig = new LogConfig(logConfigProperties(maxMessageSize = maxMessageSize, segmentSize = 256))
+ log.updateConfig(logConfig)
+
+ // with compression enabled, these messages will be written as a single message containing all the individual messages
+ var appendsV0 = writeDupsSingleMessageSet(numKeys = 2, numDups = 3, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V0)
+ appendsV0 ++= writeDupsSingleMessageSet(numKeys = 2, startKey = 3, numDups = 2, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V0)
+
+ var appendsV1 = writeDupsSingleMessageSet(startKey = 4, numKeys = 2, numDups = 2, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V1)
+ appendsV1 ++= writeDupsSingleMessageSet(startKey = 4, numKeys = 2, numDups = 2, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V1)
+ appendsV1 ++= writeDupsSingleMessageSet(startKey = 6, numKeys = 2, numDups = 2, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V1)
+
+ val appends = appendsV0 ++ appendsV1
+
+ val v0RecordKeysWithNoV1V2Updates = (appendsV0.map(_._1).toSet -- appendsV1.map(_._1)).map(_.toString)
+
+ // roll the log so that all appended messages can be compacted
+ log.roll()
+ val startSize = log.size
+ cleaner.startup()
+
+ val firstDirty = log.activeSegment.baseOffset
+ assertTrue(firstDirty > appendsV0.size) // ensure we clean data from V0 and V1
Review Comment:
To ensure we clean v1 data, we need to include appendsV1.size too, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1904849231
##########
core/src/test/scala/unit/kafka/log/LogCleanerParameterizedIntegrationTest.scala:
##########
@@ -134,6 +135,131 @@ class LogCleanerParameterizedIntegrationTest extends AbstractLogCleanerIntegrati
assertEquals(toMap(messages), toMap(read), "Contents of the map shouldn't change")
}
+ @ParameterizedTest
+ @ArgumentsSource(classOf[LogCleanerParameterizedIntegrationTest.ExcludeZstd])
+ def testCleanerWithMessageFormatV0V1V2(compressionType: CompressionType): Unit = {
+ val compression = Compression.of(compressionType).build()
+ val largeMessageKey = 20
+ val (largeMessageValue, largeMessageSet) = createLargeSingleMessageSet(largeMessageKey, RecordBatch.MAGIC_VALUE_V0, compression)
+ val maxMessageSize = compression match {
+ case Compression.NONE => largeMessageSet.sizeInBytes
+ case _ =>
+ // the broker assigns absolute offsets for message format 0 which potentially causes the compressed size to
+ // increase because the broker offsets are larger than the ones assigned by the client
+ // adding `6` to the message set size is good enough for this test: it covers the increased message size while
+ // still being less than the overhead introduced by the conversion from message format version 0 to 1
+ largeMessageSet.sizeInBytes + 6
+ }
+
+ cleaner = makeCleaner(partitions = topicPartitions, maxMessageSize = maxMessageSize)
+
+ val log = cleaner.logs.get(topicPartitions(0))
+ val props = logConfigProperties(maxMessageSize = maxMessageSize)
+ props.put(TopicConfig.MESSAGE_TIMESTAMP_TYPE_CONFIG, TimestampType.LOG_APPEND_TIME.name)
+ val logConfig = new LogConfig(props)
+ log.updateConfig(logConfig)
+
+ val appends1 = writeDups(numKeys = 100, numDups = 3, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V0)
+ val startSize = log.size
+ cleaner.startup()
+
+ val firstDirty = log.activeSegment.baseOffset
+ checkLastCleaned("log", 0, firstDirty)
+ val compactedSize = log.logSegments.asScala.map(_.size).sum
+ assertTrue(startSize > compactedSize, s"log should have been compacted: startSize=$startSize compactedSize=$compactedSize")
+
+ checkLogAfterAppendingDups(log, startSize, appends1)
+
+ val dupsV0 = writeDups(numKeys = 40, numDups = 3, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V0)
+ val appendInfo = log.appendAsLeaderWithRecordVersion(largeMessageSet, leaderEpoch = 0, recordVersion = RecordVersion.V0)
+ // move LSO forward to increase compaction bound
+ log.updateHighWatermark(log.logEndOffset)
+ val largeMessageOffset = appendInfo.firstOffset
+
+ // also add some messages with version 1 and version 2 to check that we handle mixed format versions correctly
+ val dupsV1 = writeDups(startKey = 30, numKeys = 40, numDups = 3, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V1)
+ val dupsV2 = writeDups(startKey = 15, numKeys = 5, numDups = 3, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V2)
+
+ val v0RecordKeysWithNoV1V2Updates = (appends1.map(_._1).toSet -- dupsV1.map(_._1) -- dupsV2.map(_._1)).map(_.toString)
+ val appends2: Seq[(Int, String, Long)] =
+ appends1 ++ dupsV0 ++ Seq((largeMessageKey, largeMessageValue, largeMessageOffset)) ++ dupsV1 ++ dupsV2
+
+ // roll the log so that all appended messages can be compacted
+ log.roll()
+ val firstDirty2 = log.activeSegment.baseOffset
+ checkLastCleaned("log", 0, firstDirty2)
+
+ checkLogAfterAppendingDups(log, startSize, appends2)
+ checkLogAfterConvertingToV2(compressionType, log, logConfig.messageTimestampType, v0RecordKeysWithNoV1V2Updates)
+ }
+
+ @ParameterizedTest
+ @ArgumentsSource(classOf[LogCleanerParameterizedIntegrationTest.ExcludeZstd])
+ def testCleaningNestedMessagesWithV0V1(compressionType: CompressionType): Unit = {
+ val compression = Compression.of(compressionType).build()
+ val maxMessageSize = 192
+ cleaner = makeCleaner(partitions = topicPartitions, maxMessageSize = maxMessageSize, segmentSize = 256)
+
+ val log = cleaner.logs.get(topicPartitions(0))
+ val logConfig = new LogConfig(logConfigProperties(maxMessageSize = maxMessageSize, segmentSize = 256))
+ log.updateConfig(logConfig)
+
+ // with compression enabled, these messages will be written as a single message containing all the individual messages
+ var appendsV0 = writeDupsSingleMessageSet(numKeys = 2, numDups = 3, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V0)
+ appendsV0 ++= writeDupsSingleMessageSet(numKeys = 2, startKey = 3, numDups = 2, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V0)
+
+ var appendsV1 = writeDupsSingleMessageSet(startKey = 4, numKeys = 2, numDups = 2, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V1)
+ appendsV1 ++= writeDupsSingleMessageSet(startKey = 4, numKeys = 2, numDups = 2, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V1)
+ appendsV1 ++= writeDupsSingleMessageSet(startKey = 6, numKeys = 2, numDups = 2, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V1)
+
+ val appends = appendsV0 ++ appendsV1
+
+ val v0RecordKeysWithNoV1V2Updates = (appendsV0.map(_._1).toSet -- appendsV1.map(_._1)).map(_.toString)
+
+ // roll the log so that all appended messages can be compacted
+ log.roll()
+ val startSize = log.size
+ cleaner.startup()
+
+ val firstDirty = log.activeSegment.baseOffset
+ assertTrue(firstDirty > appendsV0.size) // ensure we clean data from V0 and V1
Review Comment:
Makes sense - the original code did not include the `log.roll` and that's why it was written like this. I'll make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1899169424
##########
core/src/test/scala/unit/kafka/log/LogCleanerParameterizedIntegrationTest.scala:
##########
@@ -134,6 +135,131 @@ class LogCleanerParameterizedIntegrationTest extends AbstractLogCleanerIntegrati
assertEquals(toMap(messages), toMap(read), "Contents of the map shouldn't change")
}
+ @ParameterizedTest
+ @ArgumentsSource(classOf[LogCleanerParameterizedIntegrationTest.ExcludeZstd])
+ def testCleanerWithMessageFormatV0V1V2(compressionType: CompressionType): Unit = {
Review Comment:
The tests added in this class were originally removed via #18267 - added them back, modified them to fit the new reality and extended them to include more extensive verification of the log after compaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18321:
URL: https://github.com/apache/kafka/pull/18321#issuecomment-2564796127
@junrao Would you be able to review this PR? I would like to include it in Apache Kafka 4.0, if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1899169082
##########
core/src/test/scala/unit/kafka/log/LogTestUtils.scala:
##########
@@ -262,12 +261,6 @@ object LogTestUtils {
def listProducerSnapshotOffsets(logDir: File): Seq[Long] =
ProducerStateManager.listSnapshotFiles(logDir).asScala.map(_.offset).sorted.toSeq
- def assertLeaderEpochCacheEmpty(log: UnifiedLog): Unit = {
Review Comment:
Unused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1906470117
##########
core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:
##########
@@ -264,68 +265,6 @@ class ReplicaManagerTest {
when(cache.getAliveBrokerNodes(any[ListenerName])).thenReturn(aliveBrokers)
}
- @Test
- def testMaybeAddLogDirFetchersWithoutEpochCache(): Unit = {
Review Comment:
@junrao do you agree that we don't need this test anymore? Or is there a purpose that is not made clear by the test name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1898947254
##########
core/src/test/scala/unit/kafka/log/UnifiedLogTest.scala:
##########
@@ -2580,6 +2580,23 @@ class UnifiedLogTest {
assertEquals(None, log.leaderEpochCache.flatMap(_.latestEpoch.toScala))
}
+ @Test
+ def testLeaderEpochCacheCreatedAfterMessageFormatUpgrade(): Unit = {
Review Comment:
This test was originally removed via #18267 - added it back and modified it to take into account that the leader epoch cache always exists now (since the configured record version is always V2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1906472125
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -1680,9 +1671,9 @@ public FetchDataInfo read(RemoteStorageFetchInfo remoteStorageFetchInfo) throws
OptionalInt epoch = OptionalInt.empty();
if (logOptional.isPresent()) {
- Option<LeaderEpochFileCache> leaderEpochCache = logOptional.get().leaderEpochCache();
- if (leaderEpochCache != null && leaderEpochCache.isDefined()) {
Review Comment:
@junrao It's odd that we were checking for `null` here - do you know if we can remove this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1904847796
##########
core/src/test/scala/unit/kafka/log/LogCleanerParameterizedIntegrationTest.scala:
##########
@@ -134,6 +135,131 @@ class LogCleanerParameterizedIntegrationTest extends AbstractLogCleanerIntegrati
assertEquals(toMap(messages), toMap(read), "Contents of the map shouldn't change")
}
+ @ParameterizedTest
+ @ArgumentsSource(classOf[LogCleanerParameterizedIntegrationTest.ExcludeZstd])
+ def testCleanerWithMessageFormatV0V1V2(compressionType: CompressionType): Unit = {
+ val compression = Compression.of(compressionType).build()
+ val largeMessageKey = 20
+ val (largeMessageValue, largeMessageSet) = createLargeSingleMessageSet(largeMessageKey, RecordBatch.MAGIC_VALUE_V0, compression)
+ val maxMessageSize = compression match {
+ case Compression.NONE => largeMessageSet.sizeInBytes
+ case _ =>
+ // the broker assigns absolute offsets for message format 0 which potentially causes the compressed size to
+ // increase because the broker offsets are larger than the ones assigned by the client
+ // adding `6` to the message set size is good enough for this test: it covers the increased message size while
+ // still being less than the overhead introduced by the conversion from message format version 0 to 1
+ largeMessageSet.sizeInBytes + 6
Review Comment:
Will try to remove it - this was the existing code and it may have been written before this was allowed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1909030619
##########
core/src/test/scala/unit/kafka/log/UnifiedLogTest.scala:
##########
@@ -3662,14 +3673,9 @@ class UnifiedLogTest {
assertTrue(newDir.exists())
log.renameDir(newDir.getName, false)
- assertTrue(log.leaderEpochCache.isEmpty)
+ assertFalse(log.leaderEpochCache.nonEmpty)
assertTrue(log.partitionMetadataFile.isEmpty)
assertEquals(0, log.logEndOffset)
- // verify that records appending can still succeed
- // even with the uninitialized leaderEpochCache and partitionMetadataFile
Review Comment:
We don't support appending with the uninitialized leaderEpochCache anymore - it was questionable to support this after `renameDir(reinitialize=false)` since the former is invoked as part of the deletion of the log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1905822853
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -757,7 +769,8 @@ class UnifiedLog(@volatile var logStartOffset: Long,
leaderEpoch: Int,
requestLocal: Option[RequestLocal],
verificationGuard: VerificationGuard,
- ignoreRecordSize: Boolean): LogAppendInfo = {
+ ignoreRecordSize: Boolean,
+ toMagic: Byte = RecordBatch.CURRENT_MAGIC_VALUE): LogAppendInfo = {
Review Comment:
There is some validation logic for v0/v1 records in `LogValidator`, which is potentially useful for ingesting records of old format. So, we can keep this as it is for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1904845831
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -509,8 +509,8 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
private def initializeLeaderEpochCache(): Unit = lock synchronized {
- leaderEpochCache = UnifiedLog.maybeCreateLeaderEpochCache(
- dir, topicPartition, logDirFailureChannel, logIdent, leaderEpochCache, scheduler)
+ leaderEpochCache = Some(UnifiedLog.createLeaderEpochCache(
Review Comment:
I had done that at first, but then I noticed that the following code was setting this to `None`. What would be the appropriate behavior if we remove the `Option`? Would we set it to `null`? Happy to give that a try if that makes sense to you.
```java
def renameDir(name: String, shouldReinitialize: Boolean): Unit = {
lock synchronized {
maybeHandleIOException(s"Error while renaming dir for $topicPartition in log dir ${dir.getParent}") {
// Flush partitionMetadata file before initializing again
maybeFlushMetadataFile()
if (localLog.renameDir(name)) {
producerStateManager.updateParentDir(dir)
if (shouldReinitialize) {
// re-initialize leader epoch cache so that LeaderEpochCheckpointFile.checkpoint can correctly reference
// the checkpoint file in renamed log directory
initializeLeaderEpochCache()
initializePartitionMetadata()
} else {
leaderEpochCache = None
partitionMetadataFile = None
}
}
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1907358875
##########
core/src/test/scala/unit/kafka/log/LogCleanerParameterizedIntegrationTest.scala:
##########
@@ -134,6 +135,131 @@ class LogCleanerParameterizedIntegrationTest extends AbstractLogCleanerIntegrati
assertEquals(toMap(messages), toMap(read), "Contents of the map shouldn't change")
}
+ @ParameterizedTest
+ @ArgumentsSource(classOf[LogCleanerParameterizedIntegrationTest.ExcludeZstd])
+ def testCleanerWithMessageFormatV0V1V2(compressionType: CompressionType): Unit = {
+ val compression = Compression.of(compressionType).build()
+ val largeMessageKey = 20
+ val (largeMessageValue, largeMessageSet) = createLargeSingleMessageSet(largeMessageKey, RecordBatch.MAGIC_VALUE_V0, compression)
+ val maxMessageSize = compression match {
+ case Compression.NONE => largeMessageSet.sizeInBytes
+ case _ =>
+ // the broker assigns absolute offsets for message format 0 which potentially causes the compressed size to
+ // increase because the broker offsets are larger than the ones assigned by the client
+ // adding `6` to the message set size is good enough for this test: it covers the increased message size while
+ // still being less than the overhead introduced by the conversion from message format version 0 to 1
+ largeMessageSet.sizeInBytes + 6
+ }
+
+ cleaner = makeCleaner(partitions = topicPartitions, maxMessageSize = maxMessageSize)
+
+ val log = cleaner.logs.get(topicPartitions(0))
+ val props = logConfigProperties(maxMessageSize = maxMessageSize)
+ props.put(TopicConfig.MESSAGE_TIMESTAMP_TYPE_CONFIG, TimestampType.LOG_APPEND_TIME.name)
+ val logConfig = new LogConfig(props)
+ log.updateConfig(logConfig)
+
+ val appends1 = writeDups(numKeys = 100, numDups = 3, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V0)
+ val startSize = log.size
+ cleaner.startup()
+
+ val firstDirty = log.activeSegment.baseOffset
+ checkLastCleaned("log", 0, firstDirty)
+ val compactedSize = log.logSegments.asScala.map(_.size).sum
+ assertTrue(startSize > compactedSize, s"log should have been compacted: startSize=$startSize compactedSize=$compactedSize")
+
+ checkLogAfterAppendingDups(log, startSize, appends1)
+
+ val dupsV0 = writeDups(numKeys = 40, numDups = 3, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V0)
+ val appendInfo = log.appendAsLeaderWithRecordVersion(largeMessageSet, leaderEpoch = 0, recordVersion = RecordVersion.V0)
+ // move LSO forward to increase compaction bound
+ log.updateHighWatermark(log.logEndOffset)
+ val largeMessageOffset = appendInfo.firstOffset
+
+ // also add some messages with version 1 and version 2 to check that we handle mixed format versions correctly
+ val dupsV1 = writeDups(startKey = 30, numKeys = 40, numDups = 3, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V1)
+ val dupsV2 = writeDups(startKey = 15, numKeys = 5, numDups = 3, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V2)
+
+ val v0RecordKeysWithNoV1V2Updates = (appends1.map(_._1).toSet -- dupsV1.map(_._1) -- dupsV2.map(_._1)).map(_.toString)
+ val appends2: Seq[(Int, String, Long)] =
+ appends1 ++ dupsV0 ++ Seq((largeMessageKey, largeMessageValue, largeMessageOffset)) ++ dupsV1 ++ dupsV2
+
+ // roll the log so that all appended messages can be compacted
+ log.roll()
+ val firstDirty2 = log.activeSegment.baseOffset
+ checkLastCleaned("log", 0, firstDirty2)
+
+ checkLogAfterAppendingDups(log, startSize, appends2)
+ checkLogAfterConvertingToV2(compressionType, log, logConfig.messageTimestampType, v0RecordKeysWithNoV1V2Updates)
+ }
+
+ @ParameterizedTest
+ @ArgumentsSource(classOf[LogCleanerParameterizedIntegrationTest.ExcludeZstd])
+ def testCleaningNestedMessagesWithV0V1(compressionType: CompressionType): Unit = {
+ val compression = Compression.of(compressionType).build()
+ val maxMessageSize = 192
+ cleaner = makeCleaner(partitions = topicPartitions, maxMessageSize = maxMessageSize, segmentSize = 256)
+
+ val log = cleaner.logs.get(topicPartitions(0))
+ val logConfig = new LogConfig(logConfigProperties(maxMessageSize = maxMessageSize, segmentSize = 256))
+ log.updateConfig(logConfig)
+
+ // with compression enabled, these messages will be written as a single message containing all the individual messages
+ var appendsV0 = writeDupsSingleMessageSet(numKeys = 2, numDups = 3, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V0)
+ appendsV0 ++= writeDupsSingleMessageSet(numKeys = 2, startKey = 3, numDups = 2, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V0)
+
+ var appendsV1 = writeDupsSingleMessageSet(startKey = 4, numKeys = 2, numDups = 2, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V1)
+ appendsV1 ++= writeDupsSingleMessageSet(startKey = 4, numKeys = 2, numDups = 2, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V1)
+ appendsV1 ++= writeDupsSingleMessageSet(startKey = 6, numKeys = 2, numDups = 2, log = log, codec = compression, magicValue = RecordBatch.MAGIC_VALUE_V1)
+
+ val appends = appendsV0 ++ appendsV1
+
+ val v0RecordKeysWithNoV1V2Updates = (appendsV0.map(_._1).toSet -- appendsV1.map(_._1)).map(_.toString)
+
+ // roll the log so that all appended messages can be compacted
+ log.roll()
+ val startSize = log.size
+ cleaner.startup()
+
+ val firstDirty = log.activeSegment.baseOffset
+ assertTrue(firstDirty > appendsV0.size) // ensure we clean data from V0 and V1
Review Comment:
Made this change, I had to adjust the condition to `>=` for the test to pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1907350932
##########
core/src/test/scala/unit/kafka/log/LogCleanerParameterizedIntegrationTest.scala:
##########
@@ -134,6 +135,131 @@ class LogCleanerParameterizedIntegrationTest extends AbstractLogCleanerIntegrati
assertEquals(toMap(messages), toMap(read), "Contents of the map shouldn't change")
}
+ @ParameterizedTest
+ @ArgumentsSource(classOf[LogCleanerParameterizedIntegrationTest.ExcludeZstd])
+ def testCleanerWithMessageFormatV0V1V2(compressionType: CompressionType): Unit = {
+ val compression = Compression.of(compressionType).build()
+ val largeMessageKey = 20
+ val (largeMessageValue, largeMessageSet) = createLargeSingleMessageSet(largeMessageKey, RecordBatch.MAGIC_VALUE_V0, compression)
+ val maxMessageSize = compression match {
+ case Compression.NONE => largeMessageSet.sizeInBytes
+ case _ =>
+ // the broker assigns absolute offsets for message format 0 which potentially causes the compressed size to
+ // increase because the broker offsets are larger than the ones assigned by the client
+ // adding `6` to the message set size is good enough for this test: it covers the increased message size while
+ // still being less than the overhead introduced by the conversion from message format version 0 to 1
+ largeMessageSet.sizeInBytes + 6
Review Comment:
Removing it causes the test to fail with:
> org.apache.kafka.common.errors.RecordTooLargeException: Message batch size is 194 bytes in append topartition log-0 which exceeds the maximum configured size of 190.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1905814075
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -509,8 +509,8 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
private def initializeLeaderEpochCache(): Unit = lock synchronized {
- leaderEpochCache = UnifiedLog.maybeCreateLeaderEpochCache(
- dir, topicPartition, logDirFailureChannel, logIdent, leaderEpochCache, scheduler)
+ leaderEpochCache = Some(UnifiedLog.createLeaderEpochCache(
Review Comment:
We set `shouldReinitialize` to false when the replica is no longer needed. So, we can just set `leaderEpochCache` to null in that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1907576962
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -509,8 +509,8 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
private def initializeLeaderEpochCache(): Unit = lock synchronized {
- leaderEpochCache = UnifiedLog.maybeCreateLeaderEpochCache(
- dir, topicPartition, logDirFailureChannel, logIdent, leaderEpochCache, scheduler)
+ leaderEpochCache = UnifiedLog.createLeaderEpochCache(
Review Comment:
This is an existing issue. `initializeLeaderEpochCache` is better named as `reinitializeLeaderEpochCache`.
##########
core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:
##########
@@ -264,68 +265,6 @@ class ReplicaManagerTest {
when(cache.getAliveBrokerNodes(any[ListenerName])).thenReturn(aliveBrokers)
}
- @Test
- def testMaybeAddLogDirFetchersWithoutEpochCache(): Unit = {
Review Comment:
This test is still useful. We can just leave `partition.futureLog.get.leaderEpochCache` unchanged. It will be initialized as empty and the fetcher thread will default to initializing the fetch offset with HWM. We probably want to adjust the test name accordingly.
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1768,7 +1759,9 @@ class UnifiedLog(@volatile var logStartOffset: Long,
lock synchronized {
localLog.checkIfMemoryMappedBufferClosed()
producerExpireCheck.cancel(true)
- leaderEpochCache.foreach(_.clear())
+ // `renameDir` with `shouldReinitialize=false` sets this to `null` and it's usually (but not always) called before this method
+ if (leaderEpochCache != null)
+ leaderEpochCache.clear()
Review Comment:
Yes, I agree that we can just call `clear` on `leaderEpochCache` instead of setting it to null in `renameDir`.
##########
core/src/test/scala/unit/kafka/log/LogCleanerParameterizedIntegrationTest.scala:
##########
@@ -134,6 +135,131 @@ class LogCleanerParameterizedIntegrationTest extends AbstractLogCleanerIntegrati
assertEquals(toMap(messages), toMap(read), "Contents of the map shouldn't change")
}
+ @ParameterizedTest
+ @ArgumentsSource(classOf[LogCleanerParameterizedIntegrationTest.ExcludeZstd])
+ def testCleanerWithMessageFormatV0V1V2(compressionType: CompressionType): Unit = {
+ val compression = Compression.of(compressionType).build()
+ val largeMessageKey = 20
+ val (largeMessageValue, largeMessageSet) = createLargeSingleMessageSet(largeMessageKey, RecordBatch.MAGIC_VALUE_V0, compression)
+ val maxMessageSize = compression match {
+ case Compression.NONE => largeMessageSet.sizeInBytes
+ case _ =>
+ // the broker assigns absolute offsets for message format 0 which potentially causes the compressed size to
+ // increase because the broker offsets are larger than the ones assigned by the client
+ // adding `6` to the message set size is good enough for this test: it covers the increased message size while
+ // still being less than the overhead introduced by the conversion from message format version 0 to 1
+ largeMessageSet.sizeInBytes + 6
Review Comment:
Ah. This is needed for appending the initial message to the log, not during cleaning.
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -1680,9 +1671,9 @@ public FetchDataInfo read(RemoteStorageFetchInfo remoteStorageFetchInfo) throws
OptionalInt epoch = OptionalInt.empty();
if (logOptional.isPresent()) {
- Option<LeaderEpochFileCache> leaderEpochCache = logOptional.get().leaderEpochCache();
- if (leaderEpochCache != null && leaderEpochCache.isDefined()) {
Review Comment:
Yes, we can remove this check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18321:
URL: https://github.com/apache/kafka/pull/18321#issuecomment-2581541375
@junrao Submitted the follow-up PR: #18468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1907342970
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1768,7 +1759,9 @@ class UnifiedLog(@volatile var logStartOffset: Long,
lock synchronized {
localLog.checkIfMemoryMappedBufferClosed()
producerExpireCheck.cancel(true)
- leaderEpochCache.foreach(_.clear())
+ // `renameDir` with `shouldReinitialize=false` sets this to `null` and it's usually (but not always) called before this method
+ if (leaderEpochCache != null)
+ leaderEpochCache.clear()
Review Comment:
@junrao This is interesting - do you think we should be setting the cache to `null` here too versus calling `clear` on it? It's a bit odd that we take different actions to achieve a similar outcome (clearing the in-memory state) in two different (but related) methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1907346433
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1768,7 +1759,9 @@ class UnifiedLog(@volatile var logStartOffset: Long,
lock synchronized {
localLog.checkIfMemoryMappedBufferClosed()
producerExpireCheck.cancel(true)
- leaderEpochCache.foreach(_.clear())
+ // `renameDir` with `shouldReinitialize=false` sets this to `null` and it's usually (but not always) called before this method
+ if (leaderEpochCache != null)
+ leaderEpochCache.clear()
Review Comment:
The case where we don't call `renameDir` before `delete` is during `LogManager.loadLog`.
```java
if (logDir.getName.endsWith(UnifiedLog.DeleteDirSuffix)) {
addLogToBeDeleted(log)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18321:
URL: https://github.com/apache/kafka/pull/18321#issuecomment-2580803924
@junrao Yes, good idea. Is it ok if I do that as a follow-up? It doesn't have to go into 4.0 and there are many references.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma merged PR #18321:
URL: https://github.com/apache/kafka/pull/18321
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18321:
URL: https://github.com/apache/kafka/pull/18321#issuecomment-2580574003
@junrao Addressed your comments and the tests are passing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1904846966
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -757,7 +769,8 @@ class UnifiedLog(@volatile var logStartOffset: Long,
leaderEpoch: Int,
requestLocal: Option[RequestLocal],
verificationGuard: VerificationGuard,
- ignoreRecordSize: Boolean): LogAppendInfo = {
+ ignoreRecordSize: Boolean,
+ toMagic: Byte = RecordBatch.CURRENT_MAGIC_VALUE): LogAppendInfo = {
Review Comment:
Is it worth it? I had started down that path, but I was worried that it might be too brittle. If you think it's worth it, I can file a JIRA and we can do it in trunk (I don't think it's required for 4.0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1905814075
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -509,8 +509,8 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
private def initializeLeaderEpochCache(): Unit = lock synchronized {
- leaderEpochCache = UnifiedLog.maybeCreateLeaderEpochCache(
- dir, topicPartition, logDirFailureChannel, logIdent, leaderEpochCache, scheduler)
+ leaderEpochCache = Some(UnifiedLog.createLeaderEpochCache(
Review Comment:
We set `shouldReinitialize` is to false when the replica is no longer needed. So, we can just set `leaderEpochCache` to null in that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-13093: Log compaction should write new segments with record version v2 (KIP-724) [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #18321:
URL: https://github.com/apache/kafka/pull/18321#discussion_r1907346433
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1768,7 +1759,9 @@ class UnifiedLog(@volatile var logStartOffset: Long,
lock synchronized {
localLog.checkIfMemoryMappedBufferClosed()
producerExpireCheck.cancel(true)
- leaderEpochCache.foreach(_.clear())
+ // `renameDir` with `shouldReinitialize=false` sets this to `null` and it's usually (but not always) called before this method
+ if (leaderEpochCache != null)
+ leaderEpochCache.clear()
Review Comment:
The case where we don't call `renameDir` before `delete` is
```java
if (logDir.getName.endsWith(UnifiedLog.DeleteDirSuffix)) {
addLogToBeDeleted(log)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org