You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "kevin-wu24 (via GitHub)" <gi...@apache.org> on 2024/10/22 20:33:37 UTC
[PR] KAFKA-16964: Integration tests for adding and removing voters [kafka]
kevin-wu24 opened a new pull request, #17582:
URL: https://github.com/apache/kafka/pull/17582
This PR continues on the work of https://github.com/apache/kafka/pull/16991
From discussion, the changes regarding feature levels in the previous PR is considered out of scope for this change. Also, an "add voter" test still needs to be added.
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16964: Integration tests for adding and removing voters [kafka]
Posted by "kevin-wu24 (via GitHub)" <gi...@apache.org>.
kevin-wu24 commented on PR #17582:
URL: https://github.com/apache/kafka/pull/17582#issuecomment-2450583278
> @kevin-wu24 thanks for this patch!
Thanks for the review @chia7712. I pushed a patch to address some of your comments. Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16964: Integration tests for adding and removing voters [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 merged PR #17582:
URL: https://github.com/apache/kafka/pull/17582
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16964: Integration tests for adding and removing voters [kafka]
Posted by "kevin-wu24 (via GitHub)" <gi...@apache.org>.
kevin-wu24 commented on PR #17582:
URL: https://github.com/apache/kafka/pull/17582#issuecomment-2455067620
@chia7712 apologies, conflict should be fixed now. Let me know if there's anything else, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16964: Integration tests for adding and removing voters [kafka]
Posted by "kevin-wu24 (via GitHub)" <gi...@apache.org>.
kevin-wu24 commented on code in PR #17582:
URL: https://github.com/apache/kafka/pull/17582#discussion_r1824931425
##########
core/src/main/java/kafka/server/ServerSocketFactory.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.server;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.network.Selectable;
+import org.apache.kafka.common.utils.Utils;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.net.SocketException;
+import java.nio.channels.ServerSocketChannel;
+
+public interface ServerSocketFactory {
+ ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException;
+
+ class KafkaServerSocketFactory implements ServerSocketFactory {
+ public static final KafkaServerSocketFactory INSTANCE = new KafkaServerSocketFactory();
+
+ @Override
+ public ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException {
+ ServerSocketChannel socketChannel = ServerSocketChannel.open();
+ try {
+ socketChannel.configureBlocking(false);
+ if (recvBufferSize != Selectable.USE_DEFAULT_BUFFER_SIZE) {
+ socketChannel.socket().setReceiveBufferSize(recvBufferSize);
+ }
+ socketChannel.socket().bind(socketAddress, listenBacklogSize);
+ } catch (SocketException e) {
+ Utils.closeQuietly(socketChannel, "server socket");
+ throw new KafkaException(String.format("Socket server failed to bind to %s:%d: %s.",
+ socketAddress.getHostString(), socketAddress.getPort(), e.getMessage()), e);
Review Comment:
I looked at `Throwable` and it looks like the actual detail message associated with the parameter `Throwable cause` value (in this case `e`) is not automatically included in the `KafkaException`'s detail message, so `e.getMessage()` is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16964: Integration tests for adding and removing voters [kafka]
Posted by "kevin-wu24 (via GitHub)" <gi...@apache.org>.
kevin-wu24 commented on code in PR #17582:
URL: https://github.com/apache/kafka/pull/17582#discussion_r1825013396
##########
core/src/main/scala/kafka/server/metadata/KRaftMetadataCache.scala:
##########
@@ -550,8 +550,10 @@ class KRaftMetadataCache(
override def features(): FinalizedFeatures = {
val image = _currentImage
val finalizedFeatures = new java.util.HashMap[String, java.lang.Short](image.features().finalizedVersions())
- finalizedFeatures.put(KRaftVersion.FEATURE_NAME, kraftVersionSupplier.get().featureLevel())
-
+ val kraftVersionLevel = kraftVersionSupplier.get().featureLevel()
+ if (kraftVersionLevel > 0) {
Review Comment:
Features are defaulted to level 0, so we don't need that entry in the map. Since `kraftVersion` can now have feature level 1 to support KIP-853, we should add that entry to the map.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16964: Integration tests for adding and removing voters [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17582:
URL: https://github.com/apache/kafka/pull/17582#discussion_r1823388462
##########
core/src/main/java/kafka/server/ServerSocketFactory.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.server;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.network.Selectable;
+import org.apache.kafka.common.utils.Utils;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.net.SocketException;
+import java.nio.channels.ServerSocketChannel;
+
+public interface ServerSocketFactory {
Review Comment:
Could we move it to server module?
##########
core/src/main/java/kafka/server/ServerSocketFactory.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.server;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.network.Selectable;
+import org.apache.kafka.common.utils.Utils;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.net.SocketException;
+import java.nio.channels.ServerSocketChannel;
+
+public interface ServerSocketFactory {
+ ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException;
+
+ class KafkaServerSocketFactory implements ServerSocketFactory {
+ public static final KafkaServerSocketFactory INSTANCE = new KafkaServerSocketFactory();
Review Comment:
Could we move this `INSTANCE` to `ServerSocketFactory`?
```java
public interface ServerSocketFactory {
ServerSocketFactory INSTANCE = new KafkaServerSocketFactory();
```
##########
core/src/main/scala/kafka/server/metadata/KRaftMetadataCache.scala:
##########
@@ -550,8 +550,10 @@ class KRaftMetadataCache(
override def features(): FinalizedFeatures = {
val image = _currentImage
val finalizedFeatures = new java.util.HashMap[String, java.lang.Short](image.features().finalizedVersions())
- finalizedFeatures.put(KRaftVersion.FEATURE_NAME, kraftVersionSupplier.get().featureLevel())
-
+ val kraftVersionLevel = kraftVersionSupplier.get().featureLevel()
+ if (kraftVersionLevel > 0) {
Review Comment:
Is this a kind of bug fix?
##########
core/src/main/java/kafka/server/ServerSocketFactory.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.server;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.network.Selectable;
+import org.apache.kafka.common.utils.Utils;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.net.SocketException;
+import java.nio.channels.ServerSocketChannel;
+
+public interface ServerSocketFactory {
+ ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException;
+
+ class KafkaServerSocketFactory implements ServerSocketFactory {
+ public static final KafkaServerSocketFactory INSTANCE = new KafkaServerSocketFactory();
+
+ @Override
+ public ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException {
+ ServerSocketChannel socketChannel = ServerSocketChannel.open();
+ try {
+ socketChannel.configureBlocking(false);
+ if (recvBufferSize != Selectable.USE_DEFAULT_BUFFER_SIZE) {
+ socketChannel.socket().setReceiveBufferSize(recvBufferSize);
+ }
+ socketChannel.socket().bind(socketAddress, listenBacklogSize);
+ } catch (SocketException e) {
+ Utils.closeQuietly(socketChannel, "server socket");
+ throw new KafkaException(String.format("Socket server failed to bind to %s:%d: %s.",
+ socketAddress.getHostString(), socketAddress.getPort(), e.getMessage()), e);
Review Comment:
We already set `cause=e`, so is `e.getMessage()` necessary?
##########
test-common/src/main/java/org/apache/kafka/common/test/TestKitNode.java:
##########
@@ -35,6 +37,12 @@ default Set<String> logDataDirectories() {
return initialMetaPropertiesEnsemble().logDirProps().keySet();
}
+ default Uuid metadataDirectoryId() {
+ MetaProperties props = initialMetaPropertiesEnsemble().logDirProps().get(
Review Comment:
```java
return initialMetaPropertiesEnsemble().logDirProps().get(metadataDirectory()).directoryId().get();
```
##########
test-common/src/main/java/org/apache/kafka/common/test/TestKitNodes.java:
##########
@@ -57,8 +57,15 @@ public static class Builder {
private int numBrokerNodes;
private int numDisksPerBroker = 1;
private Map<Integer, Map<String, String>> perServerProperties = Collections.emptyMap();
- private BootstrapMetadata bootstrapMetadata = BootstrapMetadata.
- fromVersion(MetadataVersion.latestTesting(), "testkit");
+ private BootstrapMetadata bootstrapMetadata;
+
+ public Builder() {
+ this(BootstrapMetadata.fromVersion(MetadataVersion.latestTesting(), "testkit"));
+ }
+
+ public Builder(BootstrapMetadata bootstrapMetadata) {
Review Comment:
It seems that this new public constructor is not actually being used.
##########
test-common/src/main/java/org/apache/kafka/common/test/PreboundSocketFactoryManager.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.common.test;
+
+import kafka.server.ServerSocketFactory;
+
+import org.apache.kafka.common.utils.Utils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.nio.channels.ServerSocketChannel;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+public class PreboundSocketFactoryManager implements AutoCloseable {
+ private static final Logger LOG = LoggerFactory.getLogger(PreboundSocketFactoryManager.class);
Review Comment:
`LOG` is unused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16964: Integration tests for adding and removing voters [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #17582:
URL: https://github.com/apache/kafka/pull/17582#issuecomment-2453055607
@kevin-wu24 could you please fix the conflicts? thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16964: Integration tests for adding and removing voters [kafka]
Posted by "kevin-wu24 (via GitHub)" <gi...@apache.org>.
kevin-wu24 commented on code in PR #17582:
URL: https://github.com/apache/kafka/pull/17582#discussion_r1824999429
##########
test-common/src/main/java/org/apache/kafka/common/test/TestKitNodes.java:
##########
@@ -57,8 +57,15 @@ public static class Builder {
private int numBrokerNodes;
private int numDisksPerBroker = 1;
private Map<Integer, Map<String, String>> perServerProperties = Collections.emptyMap();
- private BootstrapMetadata bootstrapMetadata = BootstrapMetadata.
- fromVersion(MetadataVersion.latestTesting(), "testkit");
+ private BootstrapMetadata bootstrapMetadata;
+
+ public Builder() {
+ this(BootstrapMetadata.fromVersion(MetadataVersion.latestTesting(), "testkit"));
+ }
+
+ public Builder(BootstrapMetadata bootstrapMetadata) {
Review Comment:
I see a usage for `Builder(BootstrapMetadata bootstrapMetadata)` in `KRaftClusterTest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org