You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/12/17 19:11:03 UTC
[GitHub] [kafka] ijuma opened a new pull request, #13011: KAFKA-14521: Replace BrokerCompressionCodec with BrokerCompressionType
ijuma opened a new pull request, #13011:
URL: https://github.com/apache/kafka/pull/13011
This is a requirement for:
* KAFKA-14477: Move LogValidator to storage module.
For broader context on this change, please check:
* KAFKA-14470: Move log layer to storage module
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a diff in pull request #13011: KAFKA-14521: Replace BrokerCompressionCodec with BrokerCompressionType
Posted by GitBox <gi...@apache.org>.
ijuma commented on code in PR #13011:
URL: https://github.com/apache/kafka/pull/13011#discussion_r1051444671
##########
core/src/main/scala/kafka/utils/VerifiableProperties.scala:
##########
@@ -204,24 +203,6 @@ class VerifiableProperties(val props: Properties) extends Logging {
}
}
- /**
- * Parse compression codec from a property list in either. Codecs may be specified as integers, or as strings.
- * See [[kafka.message.CompressionCodec]] for more details.
- * @param name The property name
- * @param default Default compression codec
- * @return compression codec
- */
- def getCompressionCodec(name: String, default: CompressionCodec) = {
Review Comment:
This was not used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma merged pull request #13011: KAFKA-14521: Replace BrokerCompressionCodec with BrokerCompressionType
Posted by GitBox <gi...@apache.org>.
ijuma merged PR #13011:
URL: https://github.com/apache/kafka/pull/13011
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a diff in pull request #13011: KAFKA-14521: Replace BrokerCompressionCodec with BrokerCompressionType
Posted by GitBox <gi...@apache.org>.
ijuma commented on code in PR #13011:
URL: https://github.com/apache/kafka/pull/13011#discussion_r1051444671
##########
core/src/main/scala/kafka/utils/VerifiableProperties.scala:
##########
@@ -204,24 +203,6 @@ class VerifiableProperties(val props: Properties) extends Logging {
}
}
- /**
- * Parse compression codec from a property list in either. Codecs may be specified as integers, or as strings.
- * See [[kafka.message.CompressionCodec]] for more details.
- * @param name The property name
- * @param default Default compression codec
- * @return compression codec
- */
- def getCompressionCodec(name: String, default: CompressionCodec) = {
Review Comment:
This was/is unused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #13011: KAFKA-14521: Replace BrokerCompressionCodec with BrokerCompressionType
Posted by GitBox <gi...@apache.org>.
ijuma commented on PR #13011:
URL: https://github.com/apache/kafka/pull/13011#issuecomment-1356397544
Another one for your queue if you have the cycles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a diff in pull request #13011: KAFKA-14521: Replace BrokerCompressionCodec with BrokerCompressionType
Posted by GitBox <gi...@apache.org>.
ijuma commented on code in PR #13011:
URL: https://github.com/apache/kafka/pull/13011#discussion_r1053421666
##########
server-common/src/main/java/org/apache/kafka/server/record/BrokerCompressionType.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.server.record;
+
+import org.apache.kafka.common.record.CompressionType;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.stream.Collectors;
+
+import static java.util.Arrays.asList;
+
+public enum BrokerCompressionType {
+ UNCOMPRESSED("uncompressed") {
+ @Override
+ public CompressionType targetCompressionType(CompressionType producerCompressionType) {
+ return CompressionType.NONE;
+ }
+ },
+ ZSTD("zstd") {
+ @Override
+ public CompressionType targetCompressionType(CompressionType producerCompressionType) {
+ return CompressionType.ZSTD;
+ }
+ },
+ LZ4("lz4") {
+ @Override
+ public CompressionType targetCompressionType(CompressionType producerCompressionType) {
+ return CompressionType.LZ4;
+ }
+ },
+ SNAPPY("snappy") {
+ @Override
+ public CompressionType targetCompressionType(CompressionType producerCompressionType) {
+ return CompressionType.SNAPPY;
+ }
+ },
+ GZIP("gzip") {
+ @Override
+ public CompressionType targetCompressionType(CompressionType producerCompressionType) {
+ return CompressionType.GZIP;
+ }
+ },
+ PRODUCER("producer") {
+ @Override
+ public CompressionType targetCompressionType(CompressionType producerCompressionType) {
+ return producerCompressionType;
+ }
+ };
+
+ private static final List<BrokerCompressionType> VALUES = asList(values());
+
+ public final String name;
+
+ BrokerCompressionType(String name) {
+ this.name = name;
+ }
+
+ public abstract CompressionType targetCompressionType(CompressionType producerCompressionType);
Review Comment:
@dengziming Thanks, added a test.
For additional context, the plan is to move all existing tests from Scala to Java at the end. That reduces the risk of introducing a regression that gets missed by converting both tests and main code at the same time. I didn't find a unit test for `BrokerCompressionCodec`, so improving coverage in the way you described makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a diff in pull request #13011: KAFKA-14521: Replace BrokerCompressionCodec with BrokerCompressionType
Posted by GitBox <gi...@apache.org>.
ijuma commented on code in PR #13011:
URL: https://github.com/apache/kafka/pull/13011#discussion_r1051500260
##########
core/src/main/scala/kafka/server/KafkaConfig.scala:
##########
@@ -2138,8 +2138,6 @@ class KafkaConfig private(doLog: Boolean, val props: java.util.Map[_, _], dynami
" to prevent frequent changes in ISR")
require(offsetCommitRequiredAcks >= -1 && offsetCommitRequiredAcks <= offsetsTopicReplicationFactor,
"offsets.commit.required.acks must be greater or equal -1 and less or equal to offsets.topic.replication.factor")
- require(BrokerCompressionCodec.isValid(compressionType), "compression.type : " + compressionType + " is not valid." +
Review Comment:
This is handled by the validator defined in https://github.com/apache/kafka/pull/13011/files#diff-cbe6a8b71b05ed22cf09d97591225b588e9fca6caaf95d3b34a43262cfd23aa6R1316
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dengziming commented on a diff in pull request #13011: KAFKA-14521: Replace BrokerCompressionCodec with BrokerCompressionType
Posted by GitBox <gi...@apache.org>.
dengziming commented on code in PR #13011:
URL: https://github.com/apache/kafka/pull/13011#discussion_r1052852456
##########
server-common/src/main/java/org/apache/kafka/server/record/BrokerCompressionType.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.server.record;
+
+import org.apache.kafka.common.record.CompressionType;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.stream.Collectors;
+
+import static java.util.Arrays.asList;
+
+public enum BrokerCompressionType {
+ UNCOMPRESSED("uncompressed") {
+ @Override
+ public CompressionType targetCompressionType(CompressionType producerCompressionType) {
+ return CompressionType.NONE;
+ }
+ },
+ ZSTD("zstd") {
+ @Override
+ public CompressionType targetCompressionType(CompressionType producerCompressionType) {
+ return CompressionType.ZSTD;
+ }
+ },
+ LZ4("lz4") {
+ @Override
+ public CompressionType targetCompressionType(CompressionType producerCompressionType) {
+ return CompressionType.LZ4;
+ }
+ },
+ SNAPPY("snappy") {
+ @Override
+ public CompressionType targetCompressionType(CompressionType producerCompressionType) {
+ return CompressionType.SNAPPY;
+ }
+ },
+ GZIP("gzip") {
+ @Override
+ public CompressionType targetCompressionType(CompressionType producerCompressionType) {
+ return CompressionType.GZIP;
+ }
+ },
+ PRODUCER("producer") {
+ @Override
+ public CompressionType targetCompressionType(CompressionType producerCompressionType) {
+ return producerCompressionType;
+ }
+ };
+
+ private static final List<BrokerCompressionType> VALUES = asList(values());
+
+ public final String name;
+
+ BrokerCompressionType(String name) {
+ this.name = name;
+ }
+
+ public abstract CompressionType targetCompressionType(CompressionType producerCompressionType);
Review Comment:
I think there must be some tests that can cover this change, but it would be nice if we can add a test to verify this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #13011: KAFKA-14521: Replace BrokerCompressionCodec with BrokerCompressionType
Posted by GitBox <gi...@apache.org>.
ijuma commented on PR #13011:
URL: https://github.com/apache/kafka/pull/13011#issuecomment-1360076891
JDK 8 build passed, the other failures are unrelated.
@dengziming I will go ahead and merge since you approved, but please do let me know if you have additional comments with regards to the test added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org