You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "yunchipang (via GitHub)" <gi...@apache.org> on 2025/08/22 00:18:45 UTC
[PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
yunchipang opened a new pull request, #20396:
URL: https://github.com/apache/kafka/pull/20396
issue: [KAFKA-19616](https://issues.apache.org/jira/browse/KAFKA-19616)
**why**: validate log compaction works correctly with compressed data.
**what**: adds compression config options to `LogCompactionTester` tool and extends test coverage to validate log compaction with different compression types and levels.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "yunchipang (via GitHub)" <gi...@apache.org>.
yunchipang commented on code in PR #20396:
URL: https://github.com/apache/kafka/pull/20396#discussion_r2308815855
##########
tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java:
##########
@@ -240,13 +250,16 @@ public static void main(String[] args) throws Exception {
CommandLineUtils.checkRequiredArgs(parser, optionSet, options.brokerOpt, options.numMessagesOpt);
long messages = optionSet.valueOf(options.numMessagesOpt);
- String compressionType = optionSet.valueOf(options.messageCompressionOpt);
+ CompressionType compressionType = CompressionType.forName(optionSet.valueOf(options.messageCompressionOpt));
+ Integer compressionLevel = optionSet.valueOf(options.compressionLevelOpt);
int percentDeletes = optionSet.valueOf(options.percentDeletesOpt);
int dups = optionSet.valueOf(options.numDupsOpt);
String brokerUrl = optionSet.valueOf(options.brokerOpt);
int topicCount = optionSet.valueOf(options.topicsOpt);
int sleepSecs = optionSet.valueOf(options.sleepSecsOpt);
+ CompressionType.forName(compressionType.name.toLowerCase(Locale.ROOT));
Review Comment:
good catch. fixed. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "yunchipang (via GitHub)" <gi...@apache.org>.
yunchipang commented on code in PR #20396:
URL: https://github.com/apache/kafka/pull/20396#discussion_r2302386756
##########
tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java:
##########
@@ -256,7 +270,8 @@ public static void main(String[] args) throws Exception {
System.out.println("Producing " + messages + " messages..to topics " + String.join(",", topics));
Path producedDataFilePath = produceMessages(
brokerUrl, topics, messages,
- compressionType, dups, percentDeletes);
+ compressionType, compressionLevel,
Review Comment:
understood. fixed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "yunchipang (via GitHub)" <gi...@apache.org>.
yunchipang commented on PR #20396:
URL: https://github.com/apache/kafka/pull/20396#issuecomment-3230742300
running `TC_PATHS="tests/kafkatest/tests/tools/log_compaction_test.py" /bin/bash tests/docker/run_tests.sh`
```
================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.12.0
session_id: 2025-08-27--001
run time: 7 minutes 46.138 seconds
tests run: 8
passed: 8
flaky: 0
failed: 0
ignored: 0
================================================================================
test_id: kafkatest.tests.tools.log_compaction_test.LogCompactionTest.test_log_compaction.metadata_quorum=ISOLATED_KRAFT.compression_config=
status: PASS
run time: 59.595 seconds
--------------------------------------------------------------------------------
test_id: kafkatest.tests.tools.log_compaction_test.LogCompactionTest.test_log_compaction.metadata_quorum=ISOLATED_KRAFT.compression_config=.type.gzip.level.1
status: PASS
run time: 57.970 seconds
--------------------------------------------------------------------------------
test_id: kafkatest.tests.tools.log_compaction_test.LogCompactionTest.test_log_compaction.metadata_quorum=ISOLATED_KRAFT.compression_config=.type.gzip.level.9
status: PASS
run time: 57.639 seconds
--------------------------------------------------------------------------------
test_id: kafkatest.tests.tools.log_compaction_test.LogCompactionTest.test_log_compaction.metadata_quorum=ISOLATED_KRAFT.compression_config=.type.lz4.level.1
status: PASS
run time: 57.287 seconds
--------------------------------------------------------------------------------
test_id: kafkatest.tests.tools.log_compaction_test.LogCompactionTest.test_log_compaction.metadata_quorum=ISOLATED_KRAFT.compression_config=.type.lz4.level.10
status: PASS
run time: 57.778 seconds
--------------------------------------------------------------------------------
test_id: kafkatest.tests.tools.log_compaction_test.LogCompactionTest.test_log_compaction.metadata_quorum=ISOLATED_KRAFT.compression_config=.type.snappy
status: PASS
run time: 58.785 seconds
--------------------------------------------------------------------------------
test_id: kafkatest.tests.tools.log_compaction_test.LogCompactionTest.test_log_compaction.metadata_quorum=ISOLATED_KRAFT.compression_config=.type.zstd.level.1
status: PASS
run time: 57.755 seconds
--------------------------------------------------------------------------------
test_id: kafkatest.tests.tools.log_compaction_test.LogCompactionTest.test_log_compaction.metadata_quorum=ISOLATED_KRAFT.compression_config=.type.zstd.level.10
status: PASS
run time: 58.170 seconds
--------------------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "yunchipang (via GitHub)" <gi...@apache.org>.
yunchipang commented on code in PR #20396:
URL: https://github.com/apache/kafka/pull/20396#discussion_r2311337890
##########
tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java:
##########
@@ -395,12 +407,29 @@ private static void require(boolean requirement, String message) {
}
private static Path produceMessages(String brokerUrl, Set<String> topics, long messages,
- String compressionType, int dups, int percentDeletes) throws IOException {
- Map<String, Object> producerProps = Map.of(
- ProducerConfig.MAX_BLOCK_MS_CONFIG, String.valueOf(Long.MAX_VALUE),
- ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, brokerUrl,
- ProducerConfig.COMPRESSION_TYPE_CONFIG, compressionType
- );
+ CompressionType compressionType, Integer compressionLevel,
+ int dups, int percentDeletes) throws IOException {
+ Map<String, Object> producerProps = new HashMap<>();
+ producerProps.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, String.valueOf(Long.MAX_VALUE));
+ producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, brokerUrl);
+ producerProps.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, compressionType.name);
+
+ if (compressionLevel != null) {
+ switch (compressionType.name.toLowerCase(Locale.ROOT)) {
Review Comment:
fixed. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #20396:
URL: https://github.com/apache/kafka/pull/20396#issuecomment-3238868856
the failed test is already fixed by #20419
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 merged PR #20396:
URL: https://github.com/apache/kafka/pull/20396
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #20396:
URL: https://github.com/apache/kafka/pull/20396#issuecomment-3217240783
@yunchipang Could you please fix the conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #20396:
URL: https://github.com/apache/kafka/pull/20396#discussion_r2300232932
##########
tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java:
##########
@@ -226,6 +236,7 @@ public static String peekLine(BufferedReader reader) throws IOException {
private static final Random RANDOM = new Random();
+
Review Comment:
please remove this change
##########
tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java:
##########
@@ -395,12 +410,28 @@ private static void require(boolean requirement, String message) {
}
private static Path produceMessages(String brokerUrl, Set<String> topics, long messages,
- String compressionType, int dups, int percentDeletes) throws IOException {
- Map<String, Object> producerProps = Map.of(
- ProducerConfig.MAX_BLOCK_MS_CONFIG, String.valueOf(Long.MAX_VALUE),
- ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, brokerUrl,
- ProducerConfig.COMPRESSION_TYPE_CONFIG, compressionType
- );
+ String compressionType, Integer compressionLevel,
+ int dups, int percentDeletes) throws IOException {
+ Map<String, Object> producerProps = new HashMap<>();
+ producerProps.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, String.valueOf(Long.MAX_VALUE));
+ producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, brokerUrl);
+ producerProps.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, compressionType);
+
+ if (compressionLevel != null) {
+ switch (compressionType.toLowerCase(Locale.ROOT)) {
+ case "gzip":
+ producerProps.put(ProducerConfig.COMPRESSION_GZIP_LEVEL_CONFIG, compressionLevel);
+ break;
+ case "lz4":
+ producerProps.put(ProducerConfig.COMPRESSION_LZ4_LEVEL_CONFIG, compressionLevel);
+ break;
+ case "zstd":
+ producerProps.put(ProducerConfig.COMPRESSION_ZSTD_LEVEL_CONFIG, compressionLevel);
+ break;
+ default:
+ break;
Review Comment:
Could you please add warning message for the unused level?
##########
tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java:
##########
@@ -256,7 +270,8 @@ public static void main(String[] args) throws Exception {
System.out.println("Producing " + messages + " messages..to topics " + String.join(",", topics));
Path producedDataFilePath = produceMessages(
brokerUrl, topics, messages,
- compressionType, dups, percentDeletes);
+ compressionType, compressionLevel,
Review Comment:
`compressionType` could be changed from `string` to `CompressionType`. That also validates the string value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "yunchipang (via GitHub)" <gi...@apache.org>.
yunchipang commented on PR #20396:
URL: https://github.com/apache/kafka/pull/20396#issuecomment-3212932477
e2e test result
```
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/kafka-dev/core/build/dependant-testlibs/log4j-slf4j-impl-2.24.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/kafka-dev/tools/build/dependant-libs-2.13.16/log4j-slf4j-impl-2.24.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Producing 1000000 messages..to topics log-cleaner-test-1163149033400908362-0
Logging produce requests to /tmp/kafka-log-cleaner-produced-2674895871657186239.txt
Sleeping for 20seconds...
Consuming messages...
Logging consumed messages to /tmp/kafka-log-cleaner-consumed-6636562293354175110.txt
1000000 rows of data produced, 120154 rows of data consumed (88.0% reduction).
De-duplicating and validating output files...
Validated 90004 values, 0 mismatches.
Data verification is completed
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "yunchipang (via GitHub)" <gi...@apache.org>.
yunchipang commented on code in PR #20396:
URL: https://github.com/apache/kafka/pull/20396#discussion_r2302396180
##########
tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java:
##########
@@ -395,12 +410,28 @@ private static void require(boolean requirement, String message) {
}
private static Path produceMessages(String brokerUrl, Set<String> topics, long messages,
- String compressionType, int dups, int percentDeletes) throws IOException {
- Map<String, Object> producerProps = Map.of(
- ProducerConfig.MAX_BLOCK_MS_CONFIG, String.valueOf(Long.MAX_VALUE),
- ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, brokerUrl,
- ProducerConfig.COMPRESSION_TYPE_CONFIG, compressionType
- );
+ String compressionType, Integer compressionLevel,
+ int dups, int percentDeletes) throws IOException {
+ Map<String, Object> producerProps = new HashMap<>();
+ producerProps.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, String.valueOf(Long.MAX_VALUE));
+ producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, brokerUrl);
+ producerProps.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, compressionType);
+
+ if (compressionLevel != null) {
+ switch (compressionType.toLowerCase(Locale.ROOT)) {
+ case "gzip":
+ producerProps.put(ProducerConfig.COMPRESSION_GZIP_LEVEL_CONFIG, compressionLevel);
+ break;
+ case "lz4":
+ producerProps.put(ProducerConfig.COMPRESSION_LZ4_LEVEL_CONFIG, compressionLevel);
+ break;
+ case "zstd":
+ producerProps.put(ProducerConfig.COMPRESSION_ZSTD_LEVEL_CONFIG, compressionLevel);
+ break;
+ default:
+ break;
Review Comment:
added. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #20396:
URL: https://github.com/apache/kafka/pull/20396#discussion_r2299003609
##########
tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java:
##########
@@ -226,6 +236,56 @@ public static String peekLine(BufferedReader reader) throws IOException {
private static final Random RANDOM = new Random();
+ private static void validateCompressionConfig(String compressionType, Integer compressionLevel) {
+ validateCompressionType(compressionType);
+ validateCompressionLevel(compressionType, compressionLevel);
+ }
+
+ private static void validateCompressionType(String compressionType) {
+ Set<String> validTypes = Set.of("none", "gzip", "snappy", "lz4", "zstd");
+ if (!validTypes.contains(compressionType.toLowerCase(Locale.ROOT))) {
+ System.err.println("Invalid compression type: " + compressionType);
+ System.err.println("Valid types: " + validTypes);
+ Exit.exit(1);
+ }
+ }
+
+ private static void validateCompressionLevel(String compressionType, Integer compressionLevel) {
+ if (compressionLevel == null) {
+ return;
+ }
+
+ String normalizedType = compressionType.toLowerCase(Locale.ROOT);
+ switch (normalizedType) {
+ case "none":
+ case "snappy":
+ System.err.println("Compression level not supported for " + compressionType);
+ System.err.println("Remove --compression-level option");
+ Exit.exit(1);
+ break;
+ case "gzip":
+ validateLevelRange(compressionLevel, CompressionType.GZIP, "gzip");
+ break;
+ case "lz4":
+ validateLevelRange(compressionLevel, CompressionType.LZ4, "lz4");
+ break;
+ case "zstd":
+ validateLevelRange(compressionLevel, CompressionType.ZSTD, "zstd");
+ break;
+ default:
+ // This should never happen since compression type is validated first
+ throw new IllegalStateException("Unknown compression type: " + compressionType);
+ }
+ }
+
+ private static void validateLevelRange(int level, CompressionType type, String typeName) {
Review Comment:
this validation is good, but `CompressionType` already cover the checks. Perhaps the duplicate checks are not necessary.
##########
tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java:
##########
@@ -226,6 +236,56 @@ public static String peekLine(BufferedReader reader) throws IOException {
private static final Random RANDOM = new Random();
+ private static void validateCompressionConfig(String compressionType, Integer compressionLevel) {
+ validateCompressionType(compressionType);
+ validateCompressionLevel(compressionType, compressionLevel);
+ }
+
+ private static void validateCompressionType(String compressionType) {
Review Comment:
This could be replaced by `CompressionType.forName`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "yunchipang (via GitHub)" <gi...@apache.org>.
yunchipang commented on code in PR #20396:
URL: https://github.com/apache/kafka/pull/20396#discussion_r2299495289
##########
tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java:
##########
@@ -226,6 +236,56 @@ public static String peekLine(BufferedReader reader) throws IOException {
private static final Random RANDOM = new Random();
+ private static void validateCompressionConfig(String compressionType, Integer compressionLevel) {
+ validateCompressionType(compressionType);
+ validateCompressionLevel(compressionType, compressionLevel);
+ }
+
+ private static void validateCompressionType(String compressionType) {
Review Comment:
fixed!
##########
tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java:
##########
@@ -226,6 +236,56 @@ public static String peekLine(BufferedReader reader) throws IOException {
private static final Random RANDOM = new Random();
+ private static void validateCompressionConfig(String compressionType, Integer compressionLevel) {
+ validateCompressionType(compressionType);
+ validateCompressionLevel(compressionType, compressionLevel);
+ }
+
+ private static void validateCompressionType(String compressionType) {
+ Set<String> validTypes = Set.of("none", "gzip", "snappy", "lz4", "zstd");
+ if (!validTypes.contains(compressionType.toLowerCase(Locale.ROOT))) {
+ System.err.println("Invalid compression type: " + compressionType);
+ System.err.println("Valid types: " + validTypes);
+ Exit.exit(1);
+ }
+ }
+
+ private static void validateCompressionLevel(String compressionType, Integer compressionLevel) {
+ if (compressionLevel == null) {
+ return;
+ }
+
+ String normalizedType = compressionType.toLowerCase(Locale.ROOT);
+ switch (normalizedType) {
+ case "none":
+ case "snappy":
+ System.err.println("Compression level not supported for " + compressionType);
+ System.err.println("Remove --compression-level option");
+ Exit.exit(1);
+ break;
+ case "gzip":
+ validateLevelRange(compressionLevel, CompressionType.GZIP, "gzip");
+ break;
+ case "lz4":
+ validateLevelRange(compressionLevel, CompressionType.LZ4, "lz4");
+ break;
+ case "zstd":
+ validateLevelRange(compressionLevel, CompressionType.ZSTD, "zstd");
+ break;
+ default:
+ // This should never happen since compression type is validated first
+ throw new IllegalStateException("Unknown compression type: " + compressionType);
+ }
+ }
+
+ private static void validateLevelRange(int level, CompressionType type, String typeName) {
Review Comment:
fixed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "yunchipang (via GitHub)" <gi...@apache.org>.
yunchipang commented on PR #20396:
URL: https://github.com/apache/kafka/pull/20396#issuecomment-3226429394
e2e test result
```
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/kafka-dev/core/build/dependant-testlibs/log4j-slf4j-impl-2.25.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/kafka-dev/tools/build/dependant-libs-2.13.16/log4j-slf4j-impl-2.25.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Producing 1000000 messages..to topics log-cleaner-test-1932846938360647426-0
Logging produce requests to /tmp/kafka-log-cleaner-produced-4893708329550575535.txt
Sleeping for 20seconds...
Consuming messages...
Logging consumed messages to /tmp/kafka-log-cleaner-consumed-13046421783240844934.txt
1000000 rows of data produced, 120164 rows of data consumed (88.0% reduction).
De-duplicating and validating output files...
Validated 89942 values, 0 mismatches.
Data verification is completed
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "yunchipang (via GitHub)" <gi...@apache.org>.
yunchipang commented on code in PR #20396:
URL: https://github.com/apache/kafka/pull/20396#discussion_r2302383541
##########
tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java:
##########
@@ -226,6 +236,7 @@ public static String peekLine(BufferedReader reader) throws IOException {
private static final Random RANDOM = new Random();
+
Review Comment:
fixed! thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #20396:
URL: https://github.com/apache/kafka/pull/20396#discussion_r2307348317
##########
tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java:
##########
@@ -240,13 +250,16 @@ public static void main(String[] args) throws Exception {
CommandLineUtils.checkRequiredArgs(parser, optionSet, options.brokerOpt, options.numMessagesOpt);
long messages = optionSet.valueOf(options.numMessagesOpt);
- String compressionType = optionSet.valueOf(options.messageCompressionOpt);
+ CompressionType compressionType = CompressionType.forName(optionSet.valueOf(options.messageCompressionOpt));
+ Integer compressionLevel = optionSet.valueOf(options.compressionLevelOpt);
int percentDeletes = optionSet.valueOf(options.percentDeletesOpt);
int dups = optionSet.valueOf(options.numDupsOpt);
String brokerUrl = optionSet.valueOf(options.brokerOpt);
int topicCount = optionSet.valueOf(options.topicsOpt);
int sleepSecs = optionSet.valueOf(options.sleepSecsOpt);
+ CompressionType.forName(compressionType.name.toLowerCase(Locale.ROOT));
Review Comment:
this is unnecessary, right? line#253 has covered it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19616: Add compression type and level support to LogCompactionTester [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #20396:
URL: https://github.com/apache/kafka/pull/20396#discussion_r2311260676
##########
tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java:
##########
@@ -395,12 +407,29 @@ private static void require(boolean requirement, String message) {
}
private static Path produceMessages(String brokerUrl, Set<String> topics, long messages,
- String compressionType, int dups, int percentDeletes) throws IOException {
- Map<String, Object> producerProps = Map.of(
- ProducerConfig.MAX_BLOCK_MS_CONFIG, String.valueOf(Long.MAX_VALUE),
- ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, brokerUrl,
- ProducerConfig.COMPRESSION_TYPE_CONFIG, compressionType
- );
+ CompressionType compressionType, Integer compressionLevel,
+ int dups, int percentDeletes) throws IOException {
+ Map<String, Object> producerProps = new HashMap<>();
+ producerProps.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, String.valueOf(Long.MAX_VALUE));
+ producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, brokerUrl);
+ producerProps.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, compressionType.name);
+
+ if (compressionLevel != null) {
+ switch (compressionType.name.toLowerCase(Locale.ROOT)) {
Review Comment:
```java
if (compressionLevel != null) {
switch (compressionType) {
case GZIP -> producerProps.put(ProducerConfig.COMPRESSION_GZIP_LEVEL_CONFIG, compressionLevel);
case LZ4 -> producerProps.put(ProducerConfig.COMPRESSION_LZ4_LEVEL_CONFIG, compressionLevel);
case ZSTD -> producerProps.put(ProducerConfig.COMPRESSION_ZSTD_LEVEL_CONFIG, compressionLevel);
default -> System.out.println("Warning: Compression level " + compressionLevel + " is ignored for compression type "
+ compressionType.name + ". Only gzip, lz4, and zstd support compression levels.");
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org