You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "see-quick (via GitHub)" <gi...@apache.org> on 2026/03/06 10:14:31 UTC

[PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

see-quick opened a new pull request, #21657:
URL: https://github.com/apache/kafka/pull/21657

   This PR is last/final which concludes the migration of AbstractLogCleanerIntegrationTest.scala and its sub test suites merging into one compact test suite.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison merged PR #21657:
URL: https://github.com/apache/kafka/pull/21657


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #21657:
URL: https://github.com/apache/kafka/pull/21657#discussion_r2964794112


##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########
@@ -524,6 +917,56 @@ private LogCleaner makeCleaner(Iterable<TopicPartition> partitions,
             new Properties());
     }
 
+    private LogCleaner makeCleaner(Iterable<TopicPartition> partitions,

Review Comment:
   Created https://issues.apache.org/jira/browse/KAFKA-20343



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on PR #21657:
URL: https://github.com/apache/kafka/pull/21657#issuecomment-4108962282

   Can you review this, please? @chia7712 @mimaison ^^


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on code in PR #21657:
URL: https://github.com/apache/kafka/pull/21657#discussion_r2982368211


##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########
@@ -99,6 +108,7 @@ public class LogCleanerLagIntegrationTest {
 
     public record KeyValueOffset(int key, String value, long firstOffset) { }
     public record ValueAndRecords(String value, MemoryRecords records) { }
+    private record LogAndMessages(UnifiedLog log, List<KeyValueOffset> messages) { }

Review Comment:
   Can `KeyValueOffset` and `ValueAndRecords` be `private` too?



##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########
@@ -290,6 +300,388 @@ public void testIsThreadFailed() throws Exception {
         assertEquals(cleaner.cleaners().size(), cleaner.deadThreadCount());
     }
 
+    @ParameterizedTest
+    @EnumSource(CompressionType.class)
+    public void testCleanerCompaction(CompressionType compressionType) throws Exception {
+        Compression codec = Compression.of(compressionType).build();
+        int largeMessageKey = 20;
+        ValueAndRecords largeMessage = createLargeSingleMessageSet(largeMessageKey, RecordBatch.CURRENT_MAGIC_VALUE, codec);
+        int maxMessageSize = largeMessage.records().sizeInBytes();
+
+        cleaner = makeCleaner(TOPIC_PARTITIONS, maxMessageSize, 15000L);
+        UnifiedLog theLog = cleaner.logs().get(TOPIC_PARTITIONS.get(0));
+
+        List<KeyValueOffset> appends = writeDups(100, 3, theLog, codec);
+        long startSize = theLog.size();
+        cleaner.startup();
+
+        long firstDirty = theLog.activeSegment().baseOffset();
+        checkLastCleaned("log", 0, firstDirty);
+        long compactedSize = theLog.logSegments().stream().mapToLong(LogSegment::size).sum();
+        assertTrue(startSize > compactedSize,
+            "log should have been compacted: startSize=" + startSize + " compactedSize=" + compactedSize);
+
+        checkLogAfterAppendingDups(theLog, startSize, appends);
+
+        LogAppendInfo appendInfo = theLog.appendAsLeader(largeMessage.records(), 0);
+        // move LSO forward to increase compaction bound
+        theLog.updateHighWatermark(theLog.logEndOffset());
+        long largeMessageOffset = appendInfo.firstOffset();
+
+        List<KeyValueOffset> dups = writeDups(100, 3, theLog, codec, largeMessageKey + 1, RecordBatch.CURRENT_MAGIC_VALUE);
+        List<KeyValueOffset> appends2 = new ArrayList<>(appends);
+        appends2.add(new KeyValueOffset(largeMessageKey, largeMessage.value(), largeMessageOffset));
+        appends2.addAll(dups);
+        long firstDirty2 = theLog.activeSegment().baseOffset();
+        checkLastCleaned("log", 0, firstDirty2);
+
+        checkLogAfterAppendingDups(theLog, startSize, appends2);
+
+        // simulate deleting a partition, by removing it from logs
+        // force a checkpoint
+        // and make sure its gone from checkpoint file
+        cleaner.logs().remove(TOPIC_PARTITIONS.get(0));
+        cleaner.updateCheckpoints(logDir, Optional.of(TOPIC_PARTITIONS.get(0)));
+        Map<TopicPartition, Long> checkpoints = new OffsetCheckpointFile(
+            new File(logDir, LogCleanerManager.OFFSET_CHECKPOINT_FILE), null).read();
+        // we expect partition 0 to be gone
+        assertFalse(checkpoints.containsKey(TOPIC_PARTITIONS.get(0)));
+    }
+
+    @ParameterizedTest
+    @EnumSource(CompressionType.class)
+    public void testCleansCombinedCompactAndDeleteTopic(CompressionType compressionType) throws Exception {
+        Properties logProps = new Properties();
+        int retentionMs = 100000;
+        logProps.put(TopicConfig.RETENTION_MS_CONFIG, retentionMs);
+        logProps.put(TopicConfig.CLEANUP_POLICY_CONFIG, "compact,delete");
+
+        LogAndMessages result1 = runCleanerAndCheckCompacted(100, compressionType, logProps);
+        UnifiedLog theLog = result1.log();
+
+        // Set the last modified time to an old value to force deletion of old segments
+        long endOffset = theLog.logEndOffset();

Review Comment:
   I think we can inline that variable



##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########
@@ -594,11 +1028,16 @@ private ValueAndRecords createLargeSingleMessageSet(int key, byte messageFormatV
         return new ValueAndRecords(value, records);
     }
 
-    private void closeLog(UnifiedLog log) throws IOException {
+    private void closeLog(UnifiedLog log) {
         log.close();
         logs.remove(log);
     }
 
+    @BeforeEach

Review Comment:
   Having `@BeforeEach` and `@AfterEach` at the end of the file is unusual



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #21657:
URL: https://github.com/apache/kafka/pull/21657#discussion_r2993267503


##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########
@@ -594,11 +1028,16 @@ private ValueAndRecords createLargeSingleMessageSet(int key, byte messageFormatV
         return new ValueAndRecords(value, records);
     }
 
-    private void closeLog(UnifiedLog log) throws IOException {
+    private void closeLog(UnifiedLog log) {
         log.close();
         logs.remove(log);
     }
 
+    @BeforeEach

Review Comment:
   Let me move it to the top as I can see in other files so I will be consistent.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #21657:
URL: https://github.com/apache/kafka/pull/21657#discussion_r2959754647


##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########


Review Comment:
   Yeah, I can do it. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #21657:
URL: https://github.com/apache/kafka/pull/21657#discussion_r2960316192


##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########


Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on code in PR #21657:
URL: https://github.com/apache/kafka/pull/21657#discussion_r2995440465


##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########
@@ -357,19 +375,18 @@ public void testCleansCombinedCompactAndDeleteTopic(CompressionType compressionT
         logProps.put(TopicConfig.CLEANUP_POLICY_CONFIG, "compact,delete");
 
         LogAndMessages result1 = runCleanerAndCheckCompacted(100, compressionType, logProps);
-        UnifiedLog theLog = result1.log();
 
         // Set the last modified time to an old value to force deletion of old segments
-        long endOffset = theLog.logEndOffset();
-        for (LogSegment segment : theLog.logSegments()) {
+        long endOffset = result1.log().logEndOffset();

Review Comment:
   I meant inline `endOffset` not `theLog`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #21657:
URL: https://github.com/apache/kafka/pull/21657#discussion_r2995643016


##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########
@@ -357,19 +375,18 @@ public void testCleansCombinedCompactAndDeleteTopic(CompressionType compressionT
         logProps.put(TopicConfig.CLEANUP_POLICY_CONFIG, "compact,delete");
 
         LogAndMessages result1 = runCleanerAndCheckCompacted(100, compressionType, logProps);
-        UnifiedLog theLog = result1.log();
 
         // Set the last modified time to an old value to force deletion of old segments
-        long endOffset = theLog.logEndOffset();
-        for (LogSegment segment : theLog.logSegments()) {
+        long endOffset = result1.log().logEndOffset();

Review Comment:
   It was a bit confusing as I saw:
   ```java
   UnifiedLog theLog = result1.log();
   
   // Set the last modified time to an old value to force deletion of old segments
   long endOffset = theLog.logEndOffset();
   ```
   I will update in the next commit. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #21657:
URL: https://github.com/apache/kafka/pull/21657#discussion_r2894970898


##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########
@@ -290,6 +299,390 @@ public void testIsThreadFailed() throws Exception {
         assertEquals(cleaner.cleaners().size(), cleaner.deadThreadCount());
     }
 
+    @ParameterizedTest
+    @EnumSource(CompressionType.class)
+    public void testCleanerCompaction(CompressionType compressionType) throws Exception {
+        Compression codec = Compression.of(compressionType).build();
+        int largeMessageKey = 20;
+        ValueAndRecords largeMessage = createLargeSingleMessageSet(largeMessageKey, RecordBatch.CURRENT_MAGIC_VALUE, codec);
+        int maxMessageSize = largeMessage.records().sizeInBytes();
+
+        cleaner = makeCleaner(TOPIC_PARTITIONS, maxMessageSize, 15000L);
+        UnifiedLog theLog = cleaner.logs().get(TOPIC_PARTITIONS.get(0));
+
+        List<KeyValueOffset> appends = writeDups(100, 3, theLog, codec);
+        long startSize = theLog.size();
+        cleaner.startup();
+
+        long firstDirty = theLog.activeSegment().baseOffset();
+        checkLastCleaned("log", 0, firstDirty);
+        long compactedSize = theLog.logSegments().stream().mapToLong(LogSegment::size).sum();
+        assertTrue(startSize > compactedSize,
+            "log should have been compacted: startSize=" + startSize + " compactedSize=" + compactedSize);
+
+        checkLogAfterAppendingDups(theLog, startSize, appends);
+
+        LogAppendInfo appendInfo = theLog.appendAsLeader(largeMessage.records(), 0);
+        // move LSO forward to increase compaction bound
+        theLog.updateHighWatermark(theLog.logEndOffset());
+        long largeMessageOffset = appendInfo.firstOffset();
+
+        List<KeyValueOffset> dups = writeDups(100, 3, theLog, codec, largeMessageKey + 1, RecordBatch.CURRENT_MAGIC_VALUE);
+        List<KeyValueOffset> appends2 = new ArrayList<>(appends);
+        appends2.add(new KeyValueOffset(largeMessageKey, largeMessage.value(), largeMessageOffset));
+        appends2.addAll(dups);
+        long firstDirty2 = theLog.activeSegment().baseOffset();
+        checkLastCleaned("log", 0, firstDirty2);
+
+        checkLogAfterAppendingDups(theLog, startSize, appends2);
+
+        // simulate deleting a partition, by removing it from logs
+        // force a checkpoint
+        // and make sure its gone from checkpoint file
+        cleaner.logs().remove(TOPIC_PARTITIONS.get(0));
+        cleaner.updateCheckpoints(logDir, Optional.of(TOPIC_PARTITIONS.get(0)));
+        Map<TopicPartition, Long> checkpoints = new OffsetCheckpointFile(
+            new File(logDir, LogCleanerManager.OFFSET_CHECKPOINT_FILE), null).read();
+        // we expect partition 0 to be gone
+        assertFalse(checkpoints.containsKey(TOPIC_PARTITIONS.get(0)));
+    }
+
+    @ParameterizedTest
+    @EnumSource(CompressionType.class)
+    public void testCleansCombinedCompactAndDeleteTopic(CompressionType compressionType) throws Exception {
+        Properties logProps = new Properties();
+        int retentionMs = 100000;
+        logProps.put(TopicConfig.RETENTION_MS_CONFIG, retentionMs);
+        logProps.put(TopicConfig.CLEANUP_POLICY_CONFIG, "compact,delete");
+
+        LogAndMessages result1 = runCleanerAndCheckCompacted(100, compressionType, logProps);
+        UnifiedLog theLog = result1.log();
+
+        // Set the last modified time to an old value to force deletion of old segments
+        long endOffset = theLog.logEndOffset();
+        for (LogSegment segment : theLog.logSegments()) {
+            segment.setLastModified(time.milliseconds() - (2L * retentionMs));
+        }
+        TestUtils.waitForCondition(
+            () -> theLog.logStartOffset() == endOffset && theLog.numberOfSegments() == 1,
+            "Timed out waiting for deletion of old segments");
+
+        cleaner.shutdown();
+        closeLog(theLog);
+
+        // run the cleaner again to make sure if there are no issues post deletion
+        LogAndMessages result2 = runCleanerAndCheckCompacted(20, compressionType, logProps);
+
+        List<KeyValueOffset> read = readFromLogFull(result2.log());
+        assertEquals(toMap(result2.messages()), toMap(read), "Contents of the map shouldn't change");
+    }
+
+    private LogAndMessages runCleanerAndCheckCompacted(int numKeys, CompressionType compressionType,
+                                                       Properties logProps) throws Exception {
+        cleaner = makeCleaner(TOPIC_PARTITIONS.subList(0, 1), logProps, 100L);
+        UnifiedLog theLog = cleaner.logs().get(TOPIC_PARTITIONS.get(0));
+
+        List<KeyValueOffset> messages = writeDups(numKeys, 3, theLog, Compression.of(compressionType).build());
+        long startSize = theLog.size();
+
+        theLog.updateHighWatermark(theLog.logEndOffset());
+
+        long firstDirty = theLog.activeSegment().baseOffset();
+        cleaner.startup();
+
+        // should compact the log
+        checkLastCleaned("log", 0, firstDirty);
+        long compactedSize = theLog.logSegments().stream().mapToLong(LogSegment::size).sum();
+        assertTrue(startSize > compactedSize,
+            "log should have been compacted: startSize=" + startSize + " compactedSize=" + compactedSize);
+        return new LogAndMessages(theLog, messages);
+    }
+
+    @ParameterizedTest
+    @EnumSource(mode = EnumSource.Mode.EXCLUDE, names = "ZSTD")
+    public void testCleanerWithMessageFormatV0V1V2(CompressionType compressionType) throws Exception {
+        Compression compression = Compression.of(compressionType).build();
+        int largeMessageKey = 20;
+        ValueAndRecords largeMessage = createLargeSingleMessageSet(largeMessageKey, RecordBatch.MAGIC_VALUE_V0, compression);
+        int maxMessageSize;
+        if (compressionType == CompressionType.NONE) {
+            maxMessageSize = largeMessage.records().sizeInBytes();
+        } else {
+            // the broker assigns absolute offsets for message format 0 which potentially causes the compressed size to
+            // increase because the broker offsets are larger than the ones assigned by the client
+            // adding `6` to the message set size is good enough for this test: it covers the increased message size while
+            // still being less than the overhead introduced by the conversion from message format version 0 to 1
+            maxMessageSize = largeMessage.records().sizeInBytes() + 6;
+        }
+
+        cleaner = makeCleaner(TOPIC_PARTITIONS, maxMessageSize, 15000L);
+
+        UnifiedLog theLog = cleaner.logs().get(TOPIC_PARTITIONS.get(0));
+        Properties props = logConfigProperties(maxMessageSize);
+        props.put(TopicConfig.MESSAGE_TIMESTAMP_TYPE_CONFIG, TimestampType.LOG_APPEND_TIME.name);
+        LogConfig logConfig = new LogConfig(props);
+        theLog.updateConfig(logConfig);
+
+        List<KeyValueOffset> appends1 = writeDups(100, 3, theLog, compression, 0, RecordBatch.MAGIC_VALUE_V0);
+        long startSize = theLog.size();
+        cleaner.startup();
+
+        long firstDirty = theLog.activeSegment().baseOffset();
+        checkLastCleaned("log", 0, firstDirty);
+        long compactedSize = theLog.logSegments().stream().mapToLong(LogSegment::size).sum();
+        assertTrue(startSize > compactedSize,
+            "log should have been compacted: startSize=" + startSize + " compactedSize=" + compactedSize);
+
+        checkLogAfterAppendingDups(theLog, startSize, appends1);
+
+        List<KeyValueOffset> dupsV0 = writeDups(40, 3, theLog, compression, 0, RecordBatch.MAGIC_VALUE_V0);
+        LogAppendInfo appendInfo = theLog.appendAsLeaderWithRecordVersion(largeMessage.records(), 0, RecordVersion.V0);
+        // move LSO forward to increase compaction bound
+        theLog.updateHighWatermark(theLog.logEndOffset());
+        long largeMessageOffset = appendInfo.firstOffset();
+
+        // also add some messages with version 1 and version 2 to check that we handle mixed format versions correctly
+        List<KeyValueOffset> dupsV1 = writeDups(40, 3, theLog, compression, 30, RecordBatch.MAGIC_VALUE_V1);
+        List<KeyValueOffset> dupsV2 = writeDups(5, 3, theLog, compression, 15, RecordBatch.MAGIC_VALUE_V2);
+
+        Set<String> v0RecordKeysWithNoV1V2Updates = new HashSet<>();
+        Set<Integer> dupsV1Keys = new HashSet<>();
+        for (KeyValueOffset kvo : dupsV1) dupsV1Keys.add(kvo.key());
+        Set<Integer> dupsV2Keys = new HashSet<>();
+        for (KeyValueOffset kvo : dupsV2) dupsV2Keys.add(kvo.key());
+        for (KeyValueOffset kvo : appends1) {
+            if (!dupsV1Keys.contains(kvo.key()) && !dupsV2Keys.contains(kvo.key())) {
+                v0RecordKeysWithNoV1V2Updates.add(String.valueOf(kvo.key()));
+            }
+        }

Review Comment:
   I wonder if I can improve this a bit ... maybe reason why I like functional languages ... :D Maybe using streams would be better?



##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########
@@ -290,6 +299,390 @@ public void testIsThreadFailed() throws Exception {
         assertEquals(cleaner.cleaners().size(), cleaner.deadThreadCount());
     }
 
+    @ParameterizedTest
+    @EnumSource(CompressionType.class)
+    public void testCleanerCompaction(CompressionType compressionType) throws Exception {
+        Compression codec = Compression.of(compressionType).build();
+        int largeMessageKey = 20;
+        ValueAndRecords largeMessage = createLargeSingleMessageSet(largeMessageKey, RecordBatch.CURRENT_MAGIC_VALUE, codec);
+        int maxMessageSize = largeMessage.records().sizeInBytes();
+
+        cleaner = makeCleaner(TOPIC_PARTITIONS, maxMessageSize, 15000L);
+        UnifiedLog theLog = cleaner.logs().get(TOPIC_PARTITIONS.get(0));
+
+        List<KeyValueOffset> appends = writeDups(100, 3, theLog, codec);
+        long startSize = theLog.size();
+        cleaner.startup();
+
+        long firstDirty = theLog.activeSegment().baseOffset();
+        checkLastCleaned("log", 0, firstDirty);
+        long compactedSize = theLog.logSegments().stream().mapToLong(LogSegment::size).sum();
+        assertTrue(startSize > compactedSize,
+            "log should have been compacted: startSize=" + startSize + " compactedSize=" + compactedSize);
+
+        checkLogAfterAppendingDups(theLog, startSize, appends);
+
+        LogAppendInfo appendInfo = theLog.appendAsLeader(largeMessage.records(), 0);
+        // move LSO forward to increase compaction bound
+        theLog.updateHighWatermark(theLog.logEndOffset());
+        long largeMessageOffset = appendInfo.firstOffset();
+
+        List<KeyValueOffset> dups = writeDups(100, 3, theLog, codec, largeMessageKey + 1, RecordBatch.CURRENT_MAGIC_VALUE);
+        List<KeyValueOffset> appends2 = new ArrayList<>(appends);
+        appends2.add(new KeyValueOffset(largeMessageKey, largeMessage.value(), largeMessageOffset));
+        appends2.addAll(dups);
+        long firstDirty2 = theLog.activeSegment().baseOffset();
+        checkLastCleaned("log", 0, firstDirty2);
+
+        checkLogAfterAppendingDups(theLog, startSize, appends2);
+
+        // simulate deleting a partition, by removing it from logs
+        // force a checkpoint
+        // and make sure its gone from checkpoint file
+        cleaner.logs().remove(TOPIC_PARTITIONS.get(0));
+        cleaner.updateCheckpoints(logDir, Optional.of(TOPIC_PARTITIONS.get(0)));
+        Map<TopicPartition, Long> checkpoints = new OffsetCheckpointFile(
+            new File(logDir, LogCleanerManager.OFFSET_CHECKPOINT_FILE), null).read();
+        // we expect partition 0 to be gone
+        assertFalse(checkpoints.containsKey(TOPIC_PARTITIONS.get(0)));
+    }
+
+    @ParameterizedTest
+    @EnumSource(CompressionType.class)
+    public void testCleansCombinedCompactAndDeleteTopic(CompressionType compressionType) throws Exception {
+        Properties logProps = new Properties();
+        int retentionMs = 100000;
+        logProps.put(TopicConfig.RETENTION_MS_CONFIG, retentionMs);
+        logProps.put(TopicConfig.CLEANUP_POLICY_CONFIG, "compact,delete");
+
+        LogAndMessages result1 = runCleanerAndCheckCompacted(100, compressionType, logProps);
+        UnifiedLog theLog = result1.log();
+
+        // Set the last modified time to an old value to force deletion of old segments
+        long endOffset = theLog.logEndOffset();
+        for (LogSegment segment : theLog.logSegments()) {
+            segment.setLastModified(time.milliseconds() - (2L * retentionMs));
+        }
+        TestUtils.waitForCondition(
+            () -> theLog.logStartOffset() == endOffset && theLog.numberOfSegments() == 1,
+            "Timed out waiting for deletion of old segments");
+
+        cleaner.shutdown();
+        closeLog(theLog);
+
+        // run the cleaner again to make sure if there are no issues post deletion
+        LogAndMessages result2 = runCleanerAndCheckCompacted(20, compressionType, logProps);
+
+        List<KeyValueOffset> read = readFromLogFull(result2.log());
+        assertEquals(toMap(result2.messages()), toMap(read), "Contents of the map shouldn't change");
+    }
+
+    private LogAndMessages runCleanerAndCheckCompacted(int numKeys, CompressionType compressionType,
+                                                       Properties logProps) throws Exception {
+        cleaner = makeCleaner(TOPIC_PARTITIONS.subList(0, 1), logProps, 100L);
+        UnifiedLog theLog = cleaner.logs().get(TOPIC_PARTITIONS.get(0));
+
+        List<KeyValueOffset> messages = writeDups(numKeys, 3, theLog, Compression.of(compressionType).build());
+        long startSize = theLog.size();
+
+        theLog.updateHighWatermark(theLog.logEndOffset());
+
+        long firstDirty = theLog.activeSegment().baseOffset();
+        cleaner.startup();
+
+        // should compact the log
+        checkLastCleaned("log", 0, firstDirty);
+        long compactedSize = theLog.logSegments().stream().mapToLong(LogSegment::size).sum();
+        assertTrue(startSize > compactedSize,
+            "log should have been compacted: startSize=" + startSize + " compactedSize=" + compactedSize);
+        return new LogAndMessages(theLog, messages);
+    }
+
+    @ParameterizedTest
+    @EnumSource(mode = EnumSource.Mode.EXCLUDE, names = "ZSTD")
+    public void testCleanerWithMessageFormatV0V1V2(CompressionType compressionType) throws Exception {
+        Compression compression = Compression.of(compressionType).build();
+        int largeMessageKey = 20;
+        ValueAndRecords largeMessage = createLargeSingleMessageSet(largeMessageKey, RecordBatch.MAGIC_VALUE_V0, compression);
+        int maxMessageSize;
+        if (compressionType == CompressionType.NONE) {
+            maxMessageSize = largeMessage.records().sizeInBytes();
+        } else {
+            // the broker assigns absolute offsets for message format 0 which potentially causes the compressed size to
+            // increase because the broker offsets are larger than the ones assigned by the client
+            // adding `6` to the message set size is good enough for this test: it covers the increased message size while
+            // still being less than the overhead introduced by the conversion from message format version 0 to 1
+            maxMessageSize = largeMessage.records().sizeInBytes() + 6;
+        }
+
+        cleaner = makeCleaner(TOPIC_PARTITIONS, maxMessageSize, 15000L);
+
+        UnifiedLog theLog = cleaner.logs().get(TOPIC_PARTITIONS.get(0));
+        Properties props = logConfigProperties(maxMessageSize);
+        props.put(TopicConfig.MESSAGE_TIMESTAMP_TYPE_CONFIG, TimestampType.LOG_APPEND_TIME.name);
+        LogConfig logConfig = new LogConfig(props);
+        theLog.updateConfig(logConfig);
+
+        List<KeyValueOffset> appends1 = writeDups(100, 3, theLog, compression, 0, RecordBatch.MAGIC_VALUE_V0);
+        long startSize = theLog.size();
+        cleaner.startup();
+
+        long firstDirty = theLog.activeSegment().baseOffset();
+        checkLastCleaned("log", 0, firstDirty);
+        long compactedSize = theLog.logSegments().stream().mapToLong(LogSegment::size).sum();
+        assertTrue(startSize > compactedSize,
+            "log should have been compacted: startSize=" + startSize + " compactedSize=" + compactedSize);
+
+        checkLogAfterAppendingDups(theLog, startSize, appends1);
+
+        List<KeyValueOffset> dupsV0 = writeDups(40, 3, theLog, compression, 0, RecordBatch.MAGIC_VALUE_V0);
+        LogAppendInfo appendInfo = theLog.appendAsLeaderWithRecordVersion(largeMessage.records(), 0, RecordVersion.V0);
+        // move LSO forward to increase compaction bound
+        theLog.updateHighWatermark(theLog.logEndOffset());
+        long largeMessageOffset = appendInfo.firstOffset();
+
+        // also add some messages with version 1 and version 2 to check that we handle mixed format versions correctly
+        List<KeyValueOffset> dupsV1 = writeDups(40, 3, theLog, compression, 30, RecordBatch.MAGIC_VALUE_V1);
+        List<KeyValueOffset> dupsV2 = writeDups(5, 3, theLog, compression, 15, RecordBatch.MAGIC_VALUE_V2);
+
+        Set<String> v0RecordKeysWithNoV1V2Updates = new HashSet<>();
+        Set<Integer> dupsV1Keys = new HashSet<>();
+        for (KeyValueOffset kvo : dupsV1) dupsV1Keys.add(kvo.key());
+        Set<Integer> dupsV2Keys = new HashSet<>();
+        for (KeyValueOffset kvo : dupsV2) dupsV2Keys.add(kvo.key());
+        for (KeyValueOffset kvo : appends1) {
+            if (!dupsV1Keys.contains(kvo.key()) && !dupsV2Keys.contains(kvo.key())) {
+                v0RecordKeysWithNoV1V2Updates.add(String.valueOf(kvo.key()));
+            }
+        }

Review Comment:
   I wonder if I can improve this a bit ... maybe this is a reason why I like functional languages ... :D Maybe using streams would be better?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #21657:
URL: https://github.com/apache/kafka/pull/21657#discussion_r2959783843


##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########
@@ -67,8 +75,8 @@
 /**
  * This is an integration test that tests the fully integrated log cleaner
  */
-public class LogCleanerLagIntegrationTest {
-    private static final Logger log = LoggerFactory.getLogger(LogCleanerLagIntegrationTest.class);
+public class LogCleanerIntegrationTest {
+    private static final Logger log = LoggerFactory.getLogger(LogCleanerIntegrationTest.class);
 
     protected LogCleaner cleaner;
     protected final File logDir = TestUtils.tempDirectory();

Review Comment:
   Yeah, good point, thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on code in PR #21657:
URL: https://github.com/apache/kafka/pull/21657#discussion_r2959911183


##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########
@@ -524,6 +917,56 @@ private LogCleaner makeCleaner(Iterable<TopicPartition> partitions,
             new Properties());
     }
 
+    private LogCleaner makeCleaner(Iterable<TopicPartition> partitions,

Review Comment:
   I'm fine with that being a separate PR.



##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########


Review Comment:
   Initialization seems to be unnecessary now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #21657:
URL: https://github.com/apache/kafka/pull/21657#discussion_r2959719523


##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########
@@ -524,6 +917,56 @@ private LogCleaner makeCleaner(Iterable<TopicPartition> partitions,
             new Properties());
     }
 
+    private LogCleaner makeCleaner(Iterable<TopicPartition> partitions,

Review Comment:
   I did but I don't want to cause this PR increase to more lines than needed :D. I can create ISSUE that we can do such refactor later on. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #21657:
URL: https://github.com/apache/kafka/pull/21657#discussion_r2959794362


##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########
@@ -290,6 +299,390 @@ public void testIsThreadFailed() throws Exception {
         assertEquals(cleaner.cleaners().size(), cleaner.deadThreadCount());
     }
 
+    @ParameterizedTest
+    @EnumSource(CompressionType.class)
+    public void testCleanerCompaction(CompressionType compressionType) throws Exception {
+        Compression codec = Compression.of(compressionType).build();
+        int largeMessageKey = 20;
+        ValueAndRecords largeMessage = createLargeSingleMessageSet(largeMessageKey, RecordBatch.CURRENT_MAGIC_VALUE, codec);
+        int maxMessageSize = largeMessage.records().sizeInBytes();
+
+        cleaner = makeCleaner(TOPIC_PARTITIONS, maxMessageSize, 15000L);
+        UnifiedLog theLog = cleaner.logs().get(TOPIC_PARTITIONS.get(0));
+
+        List<KeyValueOffset> appends = writeDups(100, 3, theLog, codec);
+        long startSize = theLog.size();
+        cleaner.startup();
+
+        long firstDirty = theLog.activeSegment().baseOffset();
+        checkLastCleaned("log", 0, firstDirty);
+        long compactedSize = theLog.logSegments().stream().mapToLong(LogSegment::size).sum();
+        assertTrue(startSize > compactedSize,
+            "log should have been compacted: startSize=" + startSize + " compactedSize=" + compactedSize);
+
+        checkLogAfterAppendingDups(theLog, startSize, appends);
+
+        LogAppendInfo appendInfo = theLog.appendAsLeader(largeMessage.records(), 0);
+        // move LSO forward to increase compaction bound
+        theLog.updateHighWatermark(theLog.logEndOffset());
+        long largeMessageOffset = appendInfo.firstOffset();
+
+        List<KeyValueOffset> dups = writeDups(100, 3, theLog, codec, largeMessageKey + 1, RecordBatch.CURRENT_MAGIC_VALUE);
+        List<KeyValueOffset> appends2 = new ArrayList<>(appends);
+        appends2.add(new KeyValueOffset(largeMessageKey, largeMessage.value(), largeMessageOffset));
+        appends2.addAll(dups);
+        long firstDirty2 = theLog.activeSegment().baseOffset();
+        checkLastCleaned("log", 0, firstDirty2);
+
+        checkLogAfterAppendingDups(theLog, startSize, appends2);
+
+        // simulate deleting a partition, by removing it from logs
+        // force a checkpoint
+        // and make sure its gone from checkpoint file
+        cleaner.logs().remove(TOPIC_PARTITIONS.get(0));
+        cleaner.updateCheckpoints(logDir, Optional.of(TOPIC_PARTITIONS.get(0)));
+        Map<TopicPartition, Long> checkpoints = new OffsetCheckpointFile(
+            new File(logDir, LogCleanerManager.OFFSET_CHECKPOINT_FILE), null).read();
+        // we expect partition 0 to be gone
+        assertFalse(checkpoints.containsKey(TOPIC_PARTITIONS.get(0)));
+    }
+
+    @ParameterizedTest
+    @EnumSource(CompressionType.class)
+    public void testCleansCombinedCompactAndDeleteTopic(CompressionType compressionType) throws Exception {
+        Properties logProps = new Properties();
+        int retentionMs = 100000;
+        logProps.put(TopicConfig.RETENTION_MS_CONFIG, retentionMs);
+        logProps.put(TopicConfig.CLEANUP_POLICY_CONFIG, "compact,delete");
+
+        LogAndMessages result1 = runCleanerAndCheckCompacted(100, compressionType, logProps);
+        UnifiedLog theLog = result1.log();
+
+        // Set the last modified time to an old value to force deletion of old segments
+        long endOffset = theLog.logEndOffset();
+        for (LogSegment segment : theLog.logSegments()) {
+            segment.setLastModified(time.milliseconds() - (2L * retentionMs));
+        }
+        TestUtils.waitForCondition(
+            () -> theLog.logStartOffset() == endOffset && theLog.numberOfSegments() == 1,
+            "Timed out waiting for deletion of old segments");
+
+        cleaner.shutdown();
+        closeLog(theLog);
+
+        // run the cleaner again to make sure if there are no issues post deletion
+        LogAndMessages result2 = runCleanerAndCheckCompacted(20, compressionType, logProps);
+
+        List<KeyValueOffset> read = readFromLogFull(result2.log());
+        assertEquals(toMap(result2.messages()), toMap(read), "Contents of the map shouldn't change");
+    }
+
+    private LogAndMessages runCleanerAndCheckCompacted(int numKeys, CompressionType compressionType,
+                                                       Properties logProps) throws Exception {
+        cleaner = makeCleaner(TOPIC_PARTITIONS.subList(0, 1), logProps, 100L);
+        UnifiedLog theLog = cleaner.logs().get(TOPIC_PARTITIONS.get(0));
+
+        List<KeyValueOffset> messages = writeDups(numKeys, 3, theLog, Compression.of(compressionType).build());
+        long startSize = theLog.size();
+
+        theLog.updateHighWatermark(theLog.logEndOffset());
+
+        long firstDirty = theLog.activeSegment().baseOffset();
+        cleaner.startup();
+
+        // should compact the log
+        checkLastCleaned("log", 0, firstDirty);
+        long compactedSize = theLog.logSegments().stream().mapToLong(LogSegment::size).sum();
+        assertTrue(startSize > compactedSize,
+            "log should have been compacted: startSize=" + startSize + " compactedSize=" + compactedSize);
+        return new LogAndMessages(theLog, messages);
+    }
+
+    @ParameterizedTest
+    @EnumSource(mode = EnumSource.Mode.EXCLUDE, names = "ZSTD")
+    public void testCleanerWithMessageFormatV0V1V2(CompressionType compressionType) throws Exception {
+        Compression compression = Compression.of(compressionType).build();
+        int largeMessageKey = 20;
+        ValueAndRecords largeMessage = createLargeSingleMessageSet(largeMessageKey, RecordBatch.MAGIC_VALUE_V0, compression);
+        int maxMessageSize;
+        if (compressionType == CompressionType.NONE) {
+            maxMessageSize = largeMessage.records().sizeInBytes();
+        } else {
+            // the broker assigns absolute offsets for message format 0 which potentially causes the compressed size to
+            // increase because the broker offsets are larger than the ones assigned by the client
+            // adding `6` to the message set size is good enough for this test: it covers the increased message size while
+            // still being less than the overhead introduced by the conversion from message format version 0 to 1
+            maxMessageSize = largeMessage.records().sizeInBytes() + 6;
+        }
+
+        cleaner = makeCleaner(TOPIC_PARTITIONS, maxMessageSize, 15000L);
+
+        UnifiedLog theLog = cleaner.logs().get(TOPIC_PARTITIONS.get(0));
+        Properties props = logConfigProperties(maxMessageSize);
+        props.put(TopicConfig.MESSAGE_TIMESTAMP_TYPE_CONFIG, TimestampType.LOG_APPEND_TIME.name);
+        LogConfig logConfig = new LogConfig(props);
+        theLog.updateConfig(logConfig);
+
+        List<KeyValueOffset> appends1 = writeDups(100, 3, theLog, compression, 0, RecordBatch.MAGIC_VALUE_V0);
+        long startSize = theLog.size();
+        cleaner.startup();
+
+        long firstDirty = theLog.activeSegment().baseOffset();
+        checkLastCleaned("log", 0, firstDirty);
+        long compactedSize = theLog.logSegments().stream().mapToLong(LogSegment::size).sum();
+        assertTrue(startSize > compactedSize,
+            "log should have been compacted: startSize=" + startSize + " compactedSize=" + compactedSize);
+
+        checkLogAfterAppendingDups(theLog, startSize, appends1);
+
+        List<KeyValueOffset> dupsV0 = writeDups(40, 3, theLog, compression, 0, RecordBatch.MAGIC_VALUE_V0);
+        LogAppendInfo appendInfo = theLog.appendAsLeaderWithRecordVersion(largeMessage.records(), 0, RecordVersion.V0);
+        // move LSO forward to increase compaction bound
+        theLog.updateHighWatermark(theLog.logEndOffset());
+        long largeMessageOffset = appendInfo.firstOffset();
+
+        // also add some messages with version 1 and version 2 to check that we handle mixed format versions correctly
+        List<KeyValueOffset> dupsV1 = writeDups(40, 3, theLog, compression, 30, RecordBatch.MAGIC_VALUE_V1);
+        List<KeyValueOffset> dupsV2 = writeDups(5, 3, theLog, compression, 15, RecordBatch.MAGIC_VALUE_V2);
+
+        Set<String> v0RecordKeysWithNoV1V2Updates = new HashSet<>();
+        Set<Integer> dupsV1Keys = new HashSet<>();
+        for (KeyValueOffset kvo : dupsV1) dupsV1Keys.add(kvo.key());
+        Set<Integer> dupsV2Keys = new HashSet<>();
+        for (KeyValueOffset kvo : dupsV2) dupsV2Keys.add(kvo.key());
+        for (KeyValueOffset kvo : appends1) {
+            if (!dupsV1Keys.contains(kvo.key()) && !dupsV2Keys.contains(kvo.key())) {
+                v0RecordKeysWithNoV1V2Updates.add(String.valueOf(kvo.key()));
+            }
+        }
+
+        List<KeyValueOffset> appends2 = new ArrayList<>(appends1);
+        appends2.addAll(dupsV0);
+        appends2.add(new KeyValueOffset(largeMessageKey, largeMessage.value(), largeMessageOffset));
+        appends2.addAll(dupsV1);
+        appends2.addAll(dupsV2);
+
+        // roll the log so that all appended messages can be compacted
+        theLog.roll();
+        long firstDirty2 = theLog.activeSegment().baseOffset();
+        checkLastCleaned("log", 0, firstDirty2);
+
+        checkLogAfterAppendingDups(theLog, startSize, appends2);
+        checkLogAfterConvertingToV2(compressionType, theLog, logConfig.messageTimestampType, v0RecordKeysWithNoV1V2Updates);
+    }
+
+    @ParameterizedTest
+    @EnumSource(mode = EnumSource.Mode.EXCLUDE, names = "ZSTD")
+    public void testCleaningNestedMessagesWithV0V1(CompressionType compressionType) throws Exception {
+        Compression compression = Compression.of(compressionType).build();
+        int maxMessageSize = 192;
+        cleaner = makeCleaner(TOPIC_PARTITIONS, maxMessageSize, 15000L, 256);
+
+        UnifiedLog theLog = cleaner.logs().get(TOPIC_PARTITIONS.get(0));
+        LogConfig logConfig = new LogConfig(logConfigProperties(new Properties(), maxMessageSize,
+            DEFAULT_MIN_CLEANABLE_DIRTY_RATIO, DEFAULT_MIN_COMPACTION_LAG_MS,
+            DEFAULT_DELETE_DELAY, 256, DEFAULT_MAX_COMPACTION_LAG_MS));
+        theLog.updateConfig(logConfig);
+
+        // with compression enabled, these messages will be written as a single message containing all the individual messages
+        List<KeyValueOffset> appendsV0 = new ArrayList<>(writeDupsSingleMessageSet(2, 3, theLog, compression, 0, RecordBatch.MAGIC_VALUE_V0));
+        appendsV0.addAll(writeDupsSingleMessageSet(2, 2, theLog, compression, 3, RecordBatch.MAGIC_VALUE_V0));
+
+        List<KeyValueOffset> appendsV1 = new ArrayList<>(writeDupsSingleMessageSet(2, 2, theLog, compression, 4, RecordBatch.MAGIC_VALUE_V1));
+        appendsV1.addAll(writeDupsSingleMessageSet(2, 2, theLog, compression, 4, RecordBatch.MAGIC_VALUE_V1));
+        appendsV1.addAll(writeDupsSingleMessageSet(2, 2, theLog, compression, 6, RecordBatch.MAGIC_VALUE_V1));
+
+        List<KeyValueOffset> appends = new ArrayList<>(appendsV0);
+        appends.addAll(appendsV1);
+
+        Set<Integer> appendsV1Keys = new HashSet<>();
+        for (KeyValueOffset kvo : appendsV1) appendsV1Keys.add(kvo.key());
+        Set<String> v0RecordKeysWithNoV1V2Updates = new HashSet<>();
+        for (KeyValueOffset kvo : appendsV0) {
+            if (!appendsV1Keys.contains(kvo.key())) {
+                v0RecordKeysWithNoV1V2Updates.add(String.valueOf(kvo.key()));
+            }
+        }
+
+        // roll the log so that all appended messages can be compacted
+        theLog.roll();
+        long startSize = theLog.size();
+        cleaner.startup();
+
+        long firstDirty = theLog.activeSegment().baseOffset();
+        assertTrue(firstDirty >= appends.size()); // ensure we clean data from V0 and V1
+
+        checkLastCleaned("log", 0, firstDirty);
+        long compactedSize = theLog.logSegments().stream().mapToLong(LogSegment::size).sum();
+        assertTrue(startSize > compactedSize,
+            "log should have been compacted: startSize=" + startSize + " compactedSize=" + compactedSize);
+
+        checkLogAfterAppendingDups(theLog, startSize, appends);
+        checkLogAfterConvertingToV2(compressionType, theLog, logConfig.messageTimestampType, v0RecordKeysWithNoV1V2Updates);
+    }
+
+    @ParameterizedTest
+    @EnumSource(CompressionType.class)
+    public void cleanerConfigUpdateTest(CompressionType compressionType) throws Exception {
+        Compression codec = Compression.of(compressionType).build();
+        int largeMessageKey = 20;
+        ValueAndRecords largeMessage = createLargeSingleMessageSet(largeMessageKey, RecordBatch.CURRENT_MAGIC_VALUE, codec);
+        int maxMessageSize = largeMessage.records().sizeInBytes();
+
+        cleaner = makeCleaner(TOPIC_PARTITIONS, 1L, maxMessageSize, 1);
+        UnifiedLog theLog = cleaner.logs().get(TOPIC_PARTITIONS.get(0));
+
+        writeDups(100, 3, theLog, codec);
+        long startSize = theLog.size();
+        cleaner.startup();
+        assertEquals(1, cleaner.cleanerCount());
+
+        // Verify no cleaning with LogCleanerIoBufferSizeProp=1
+        long firstDirty = theLog.activeSegment().baseOffset();
+        TopicPartition topicPartition = new TopicPartition("log", 0);
+        cleaner.awaitCleaned(topicPartition, firstDirty, 10);
+        assertTrue(cleaner.cleanerManager().allCleanerCheckpoints().isEmpty(), "Should not have cleaned");
+
+        ConfigDef configDef = Utils.mergeConfigs(List.of(
+            CleanerConfig.CONFIG_DEF,
+            ServerConfigs.CONFIG_DEF
+        ));
+
+        CleanerConfig currentConfig = cleaner.currentConfig();
+        Map<String, Object> oldConfigMap = new HashMap<>();
+        oldConfigMap.put(CleanerConfig.LOG_CLEANER_THREADS_PROP, currentConfig.numThreads);
+        oldConfigMap.put(CleanerConfig.LOG_CLEANER_DEDUPE_BUFFER_SIZE_PROP, currentConfig.dedupeBufferSize);
+        oldConfigMap.put(CleanerConfig.LOG_CLEANER_DEDUPE_BUFFER_LOAD_FACTOR_PROP, currentConfig.dedupeBufferLoadFactor);
+        oldConfigMap.put(CleanerConfig.LOG_CLEANER_IO_BUFFER_SIZE_PROP, currentConfig.ioBufferSize);
+        oldConfigMap.put(ServerConfigs.MESSAGE_MAX_BYTES_CONFIG, currentConfig.maxMessageSize);
+        oldConfigMap.put(CleanerConfig.LOG_CLEANER_IO_MAX_BYTES_PER_SECOND_PROP, currentConfig.maxIoBytesPerSecond);
+        oldConfigMap.put(CleanerConfig.LOG_CLEANER_BACKOFF_MS_PROP, currentConfig.backoffMs);
+        oldConfigMap.put(CleanerConfig.LOG_CLEANER_ENABLE_PROP, currentConfig.enableCleaner);
+        AbstractConfig oldAbstractConfig = new AbstractConfig(configDef, oldConfigMap);
+
+        Map<String, Object> newConfigMap = new HashMap<>(oldConfigMap);
+        newConfigMap.put(CleanerConfig.LOG_CLEANER_THREADS_PROP, 2);
+        newConfigMap.put(CleanerConfig.LOG_CLEANER_IO_BUFFER_SIZE_PROP, 100000);
+        AbstractConfig newAbstractConfig = new AbstractConfig(configDef, newConfigMap);
+
+        cleaner.reconfigure(oldAbstractConfig, newAbstractConfig);
+
+        assertEquals(2, cleaner.cleanerCount());
+        checkLastCleaned("log", 0, firstDirty);
+        long compactedSize = theLog.logSegments().stream().mapToLong(LogSegment::size).sum();
+        assertTrue(startSize > compactedSize,
+            "log should have been compacted: startSize=" + startSize + " compactedSize=" + compactedSize);
+    }
+
+    private void checkLastCleaned(String topic, int partitionId, long firstDirty) throws InterruptedException {
+        // wait until cleaning up to base_offset, note that cleaning happens only when "log dirty ratio" is higher than
+        // TopicConfig.MIN_CLEANABLE_DIRTY_RATIO_CONFIG
+        TopicPartition topicPartition = new TopicPartition(topic, partitionId);
+        cleaner.awaitCleaned(topicPartition, firstDirty, 60000L);
+        Long lastCleaned = cleaner.cleanerManager().allCleanerCheckpoints().get(topicPartition);
+        assertTrue(lastCleaned >= firstDirty,
+            "log cleaner should have processed up to offset " + firstDirty + ", but lastCleaned=" + lastCleaned);
+    }
+
+    private void checkLogAfterAppendingDups(UnifiedLog log, long startSize, List<KeyValueOffset> appends) {
+        List<KeyValueOffset> read = readFromLogFull(log);
+        assertEquals(toMap(appends), toMap(read), "Contents of the map shouldn't change");
+        assertTrue(startSize > log.size());
+    }
+
+    private Map<Integer, KeyValueOffset> toMap(List<KeyValueOffset> messages) {
+        Map<Integer, KeyValueOffset> result = new HashMap<>();
+        for (KeyValueOffset kvo : messages) {
+            result.put(kvo.key(), kvo);
+        }
+        return result;
+    }
+
+    private List<KeyValueOffset> readFromLogFull(UnifiedLog log) {
+        List<KeyValueOffset> result = new ArrayList<>();
+        for (LogSegment segment : log.logSegments()) {
+            for (Record record : segment.log().records()) {
+                int key = Integer.parseInt(LogTestUtils.readString(record.key()));
+                String value = LogTestUtils.readString(record.value());
+                result.add(new KeyValueOffset(key, value, record.offset()));
+            }
+        }
+        return result;
+    }

Review Comment:
   Yeah, during this I actually find out that also `readKeyValuePairsFromLog` is very similar.  Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20267 Move LogCleanerParameterizedIntegrationTest to storage module [kafka]

Posted by "see-quick (via GitHub)" <gi...@apache.org>.
see-quick commented on code in PR #21657:
URL: https://github.com/apache/kafka/pull/21657#discussion_r2993259790


##########
storage/src/test/java/org/apache/kafka/storage/internals/log/LogCleanerIntegrationTest.java:
##########
@@ -99,6 +108,7 @@ public class LogCleanerLagIntegrationTest {
 
     public record KeyValueOffset(int key, String value, long firstOffset) { }
     public record ValueAndRecords(String value, MemoryRecords records) { }
+    private record LogAndMessages(UnifiedLog log, List<KeyValueOffset> messages) { }

Review Comment:
   Yeah good point, let me change it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org