You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "cmccabe (via GitHub)" <gi...@apache.org> on 2024/08/23 22:57:00 UTC
[PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
cmccabe opened a new pull request, #16991:
URL: https://github.com/apache/kafka/pull/16991
Add ReconfigurableQuorumIntegrationTest to test creating a cluster with KIP-853 enabled, to verify that the feature flag is set. Introduce functions such as TestKitNodes.setFeature that can set the level of an individual feature without replacing the whole bootstrap metadata object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1735432988
##########
core/src/main/java/kafka/server/ServerSocketFactory.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.server;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.network.Selectable;
+import org.apache.kafka.common.utils.Utils;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.net.SocketException;
+import java.nio.channels.ServerSocketChannel;
+
+public interface ServerSocketFactory {
+ ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException;
+
+ class KafkaServerSocketFactory implements ServerSocketFactory {
+ public static final KafkaServerSocketFactory INSTANCE = new KafkaServerSocketFactory();
+
+ @Override
+ public ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException {
+ ServerSocketChannel socketChannel = null;
+ try {
+ socketChannel = ServerSocketChannel.open();
Review Comment:
> there shouldn't be a resource leak in the current case where it's initialized to null, since that's handled in the catch block.
oh, you are right. thanks for sharing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on PR #16991:
URL: https://github.com/apache/kafka/pull/16991#issuecomment-2318909484
Fixed conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1735331384
##########
core/src/test/java/kafka/testkit/KafkaClusterTestKit.java:
##########
@@ -404,22 +423,44 @@ private void formatNode(
boolean writeMetadataDirectory
) {
try {
- MetaPropertiesEnsemble.Copier copier =
- new MetaPropertiesEnsemble.Copier(MetaPropertiesEnsemble.EMPTY);
- for (Entry<String, MetaProperties> entry : ensemble.logDirProps().entrySet()) {
- String logDir = entry.getKey();
- if (writeMetadataDirectory || (!ensemble.metadataLogDir().equals(Optional.of(logDir)))) {
- log.trace("Adding {} to the list of directories to format.", logDir);
- copier.setLogDirProps(logDir, entry.getValue());
+ Formatter formatter = new Formatter();
+ formatter.setNodeId(ensemble.nodeId().getAsInt());
+ formatter.setClusterId(ensemble.clusterId().get());
+ if (writeMetadataDirectory) {
+ formatter.setDirectories(ensemble.logDirProps().keySet());
+ } else {
+ formatter.setDirectories(ensemble.logDirProps().keySet().stream().
+ filter(d -> !ensemble.metadataLogDir().get().equals(d)).
+ collect(Collectors.toSet()));
+ }
+ if (formatter.directories().isEmpty()) {
+ return;
+ }
+ formatter.setReleaseVersion(nodes.bootstrapMetadata().metadataVersion());
+ formatter.setFeatureLevel(KRaftVersion.FEATURE_NAME,
+ nodes.bootstrapMetadata().featureLevel(KRaftVersion.FEATURE_NAME));
+ formatter.setUnstableFeatureVersionsEnabled(true);
+ formatter.setIgnoreFormatted(false);
+ formatter.setControllerListenerName("CONTROLLER");
+ if (writeMetadataDirectory) {
+ formatter.setMetadataLogDirectory(ensemble.metadataLogDir().get());
+ } else {
+ formatter.setMetadataLogDirectory(Optional.empty());
+ }
+ if (nodes.bootstrapMetadata().featureLevel(KRaftVersion.FEATURE_NAME) > 0) {
+ StringBuilder dynamicVotersBuilder = new StringBuilder();
+ String prefix = "";
+ for (TestKitNode controllerNode : nodes.controllerNodes().values()) {
+ int port = socketFactoryManager.
+ getOrCreatePortForListener(controllerNode.id(), "CONTROLLER");
+ dynamicVotersBuilder.append(prefix);
+ prefix = ",";
+ dynamicVotersBuilder.append(String.format("%d@localhost:%d:%s",
Review Comment:
Well, testing the parsing of voter strings is actually a good thing, that we want the tests to do. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1743091284
##########
core/src/test/java/kafka/test/junit/RaftClusterInvocationContext.java:
##########
@@ -247,52 +241,16 @@ public Map<Integer, ControllerServer> controllers() {
public void format() throws Exception {
if (formated.compareAndSet(false, true)) {
- Map<String, Features> nameToSupportedFeature = new TreeMap<>();
- Features.PRODUCTION_FEATURES.forEach(feature -> nameToSupportedFeature.put(feature.featureName(), feature));
- Map<String, Short> newFeatureLevels = new TreeMap<>();
-
- // Verify that all specified features are known to us.
- for (Map.Entry<Features, Short> entry : clusterConfig.features().entrySet()) {
- String featureName = entry.getKey().featureName();
- short level = entry.getValue();
- if (!featureName.equals(MetadataVersion.FEATURE_NAME)) {
- if (!nameToSupportedFeature.containsKey(featureName)) {
- throw new FormatterException("Unsupported feature: " + featureName +
- ". Supported features are: " + String.join(", ", nameToSupportedFeature.keySet()));
- }
- }
- newFeatureLevels.put(featureName, level);
- }
- newFeatureLevels.put(MetadataVersion.FEATURE_NAME, clusterConfig.metadataVersion().featureLevel());
-
- // Add default values for features that were not specified.
- Features.PRODUCTION_FEATURES.forEach(supportedFeature -> {
Review Comment:
`ClusterTest#features` (https://github.com/apache/kafka/blob/trunk/core/src/test/java/kafka/test/annotation/ClusterTest.java#L50) is empty by default, hence removing the code of adding default features will cause a lot of tests failed.
We can add default features to `ClusterTest`. for example:
```java
ClusterFeature[] features() default {
@ClusterFeature(feature = Features.TRANSACTION_VERSION, version = 2),
@ClusterFeature(feature = Features.GROUP_VERSION, version = 1)
};
```
The side effect is that all declarations are static so we need to update default features when we do something changes to `Features`.
Or we can keep origin behavior to add default features based on `PRODUCTION_FEATURES`. That is more flexible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on PR #16991:
URL: https://github.com/apache/kafka/pull/16991#issuecomment-2311314941
I reworked this a bit so that we can test removing voters now.
The checkstyle issue should be fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1742410514
##########
core/src/main/java/kafka/server/ServerSocketFactory.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.server;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.network.Selectable;
+import org.apache.kafka.common.utils.Utils;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.net.SocketException;
+import java.nio.channels.ServerSocketChannel;
+
+public interface ServerSocketFactory {
+ ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException;
+
+ class KafkaServerSocketFactory implements ServerSocketFactory {
+ public static final KafkaServerSocketFactory INSTANCE = new KafkaServerSocketFactory();
+
+ @Override
+ public ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException {
+ ServerSocketChannel socketChannel = ServerSocketChannel.open();
+ try {
+ socketChannel.configureBlocking(false);
+ if (recvBufferSize != Selectable.USE_DEFAULT_BUFFER_SIZE) {
+ socketChannel.socket().setReceiveBufferSize(recvBufferSize);
+ }
+ socketChannel.socket().bind(socketAddress, listenBacklogSize);
+ } catch (SocketException e) {
+ Utils.closeQuietly(socketChannel, "server socket");
+ throw new KafkaException(String.format("Socket server failed to bind to %s:%d: %s.",
Review Comment:
I did it this way to keep the same behavior as the existing `openServerSocket` function, which doesn't catch and wrap `IOException` when it's raised from `ServerSocketChannel.open()`, but does catch and wrap it when it's reaised from `socket.bind()` and subsequent calls.
If we want to change the behavior, it would be better to have a follow-on JIRA I think...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1735330570
##########
metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java:
##########
@@ -125,7 +125,7 @@ public class Formatter {
/**
* The metadata log directory.
*/
- private String metadataLogDirectory = null;
+ private Optional<String> metadataLogDirectory = null;
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #16991:
URL: https://github.com/apache/kafka/pull/16991#issuecomment-2308754564
@cmccabe could you please fix build error?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1758464571
##########
core/src/test/scala/unit/kafka/server/OffsetCommitRequestTest.scala:
##########
@@ -34,6 +35,10 @@ class OffsetCommitRequestTest(cluster: ClusterInstance) extends GroupCoordinator
serverProperties = Array(
new ClusterConfigProperty(key = GroupCoordinatorConfig.OFFSETS_TOPIC_PARTITIONS_CONFIG, value = "1"),
new ClusterConfigProperty(key = GroupCoordinatorConfig.OFFSETS_TOPIC_REPLICATION_FACTOR_CONFIG, value = "1")
+ ),
+ features = Array(
+ new ClusterFeature(feature = Features.TRANSACTION_VERSION, version = 2),
Review Comment:
this feature is unnecessary to this test case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1742405774
##########
core/src/test/java/kafka/testkit/PreboundSocketFactoryManager.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.testkit;
+
+import kafka.server.ServerSocketFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.nio.channels.ServerSocketChannel;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+public class PreboundSocketFactoryManager implements AutoCloseable {
+ private static final Logger LOG = LoggerFactory.getLogger(PreboundSocketFactoryManager.class);
+
+ private class PreboundSocketFactory implements ServerSocketFactory {
+ private final int nodeId;
+
+ private PreboundSocketFactory(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ @Override
+ public ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException {
+ ServerSocketChannel socketChannel = getSocketForListenerAndMarkAsUsed(
+ nodeId,
+ listenerName);
+ if (socketChannel != null) {
+ return socketChannel;
+ }
+ return KafkaServerSocketFactory.INSTANCE.openServerSocket(
+ listenerName,
+ socketAddress,
+ listenBacklogSize,
+ recvBufferSize);
+ }
+ }
+
+ /**
+ * True if this manager is closed.
+ */
+ private boolean closed = false;
+
+ /**
+ * Maps node IDs to socket factory objects.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, PreboundSocketFactory> factories = new HashMap<>();
+
+ /**
+ * Maps node IDs to maps of listener names to ports.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, Map<String, ServerSocketChannel>> sockets = new HashMap<>();
+
+ /**
+ * Maps node IDs to set of the listeners that were used.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, Set<String>> usedSockets = new HashMap<>();
+
+ /**
+ * Get a socket from this manager, mark it as used, and return it.
+ *
+ * @param nodeId The ID of the node.
+ * @param listener The listener for the socket.
+ *
+ * @return null if the socket was not found; the socket, otherwise.
+ */
+ public synchronized ServerSocketChannel getSocketForListenerAndMarkAsUsed(
+ int nodeId,
+ String listener
+ ) {
+ Map<String, ServerSocketChannel> socketsForNode = sockets.get(nodeId);
+ if (socketsForNode == null) {
+ return null;
+ }
+ ServerSocketChannel socket = socketsForNode.get(listener);
+ if (socket == null) {
+ return null;
+ }
+ usedSockets.computeIfAbsent(nodeId, __ -> new HashSet<>()).add(listener);
+ return socket;
+ }
+
+ /**
+ * Get or create a socket factory object associated with a given node ID.
+ *
+ * @param nodeId The ID of the node.
+ *
+ * @return The socket factory.
+ */
+ public synchronized PreboundSocketFactory getOrCreateSocketFactory(int nodeId) {
+ return factories.computeIfAbsent(nodeId, __ -> new PreboundSocketFactory(nodeId));
+ }
+
+ /**
+ * Get a specific port number. The port will be created if it does not already exist.
+ *
+ * @param nodeId The ID of the node.
+ * @param listener The listener for the socket.
+ *
+ * @return The port number.
+ */
+ public synchronized int getOrCreatePortForListener(
+ int nodeId,
+ String listener
+ ) throws IOException {
+ Map<String, ServerSocketChannel> socketsForNode =
+ sockets.computeIfAbsent(nodeId, __ -> new HashMap<>());
+ ServerSocketChannel socketChannel = socketsForNode.get(listener);
+ if (socketChannel == null) {
+ if (closed) {
+ throw new RuntimeException("Cannot open new socket: manager is closed.");
+ }
+ socketChannel = ServerSocketFactory.KafkaServerSocketFactory.INSTANCE.openServerSocket(
+ listener,
+ new InetSocketAddress(0),
+ -1,
+ -1);
+ socketsForNode.put(listener, socketChannel);
+ }
+ InetSocketAddress socketAddress = (InetSocketAddress) socketChannel.getLocalAddress();
+ return socketAddress.getPort();
+ }
+
+ @Override
+ public synchronized void close() throws Exception {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ // Close all sockets that haven't been used by a SocketServer. (We don't want to close the
+ // ones that have been used by a SocketServer because that is the responsibility of that
+ // SocketServer.)
+ for (Entry<Integer, Map<String, ServerSocketChannel>> socketsEntry : sockets.entrySet()) {
+ Set<String> usedListeners = usedSockets.getOrDefault(
+ socketsEntry.getKey(), Collections.emptySet());
+ for (Entry<String, ServerSocketChannel> entry : socketsEntry.getValue().entrySet()) {
+ if (!usedListeners.contains(entry.getKey())) {
+ try {
Review Comment:
Sorry, you're right. I misinterpreted a compiler error I got for an unrelated reason.
This should be fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1742411462
##########
core/src/test/java/kafka/testkit/KafkaClusterTestKit.java:
##########
@@ -404,22 +423,44 @@ private void formatNode(
boolean writeMetadataDirectory
) {
try {
- MetaPropertiesEnsemble.Copier copier =
- new MetaPropertiesEnsemble.Copier(MetaPropertiesEnsemble.EMPTY);
- for (Entry<String, MetaProperties> entry : ensemble.logDirProps().entrySet()) {
- String logDir = entry.getKey();
- if (writeMetadataDirectory || (!ensemble.metadataLogDir().equals(Optional.of(logDir)))) {
- log.trace("Adding {} to the list of directories to format.", logDir);
- copier.setLogDirProps(logDir, entry.getValue());
+ Formatter formatter = new Formatter();
+ formatter.setNodeId(ensemble.nodeId().getAsInt());
+ formatter.setClusterId(ensemble.clusterId().get());
+ if (writeMetadataDirectory) {
+ formatter.setDirectories(ensemble.logDirProps().keySet());
+ } else {
+ formatter.setDirectories(ensemble.logDirProps().keySet().stream().
+ filter(d -> !ensemble.metadataLogDir().get().equals(d)).
+ collect(Collectors.toSet()));
+ }
+ if (formatter.directories().isEmpty()) {
+ return;
+ }
+ formatter.setReleaseVersion(nodes.bootstrapMetadata().metadataVersion());
+ formatter.setFeatureLevel(KRaftVersion.FEATURE_NAME,
+ nodes.bootstrapMetadata().featureLevel(KRaftVersion.FEATURE_NAME));
+ formatter.setUnstableFeatureVersionsEnabled(true);
+ formatter.setIgnoreFormatted(false);
+ formatter.setControllerListenerName("CONTROLLER");
+ if (writeMetadataDirectory) {
+ formatter.setMetadataLogDirectory(ensemble.metadataLogDir().get());
+ } else {
+ formatter.setMetadataLogDirectory(Optional.empty());
Review Comment:
Sorry, I'm not sure what you are proposing here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1735324907
##########
core/src/test/java/kafka/testkit/PreboundSocketFactoryManager.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.testkit;
+
+import kafka.server.ServerSocketFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.nio.channels.ServerSocketChannel;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+public class PreboundSocketFactoryManager implements AutoCloseable {
+ private static final Logger LOG = LoggerFactory.getLogger(PreboundSocketFactoryManager.class);
+
+ private class PreboundSocketFactory implements ServerSocketFactory {
+ private final int nodeId;
+
+ private PreboundSocketFactory(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ @Override
+ public ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException {
+ ServerSocketChannel socketChannel = getSocketForListenerAndMarkAsUsed(
+ nodeId,
+ listenerName);
+ if (socketChannel != null) {
+ return socketChannel;
+ }
+ return KafkaServerSocketFactory.INSTANCE.openServerSocket(
+ listenerName,
+ socketAddress,
+ listenBacklogSize,
+ recvBufferSize);
+ }
+ }
+
+ /**
+ * True if this manager is closed.
+ */
+ private boolean closed = false;
+
+ /**
+ * Maps node IDs to socket factory objects.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, PreboundSocketFactory> factories = new HashMap<>();
+
+ /**
+ * Maps node IDs to maps of listener names to ports.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, Map<String, ServerSocketChannel>> sockets = new HashMap<>();
+
+ /**
+ * Maps node IDs to set of the listeners that were used.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, Set<String>> usedSockets = new HashMap<>();
+
+ /**
+ * Get a socket from this manager, mark it as used, and return it.
+ *
+ * @param nodeId The ID of the node.
+ * @param listener The listener for the socket.
+ *
+ * @return null if the socket was not found; the socket, otherwise.
+ */
+ public synchronized ServerSocketChannel getSocketForListenerAndMarkAsUsed(
+ int nodeId,
+ String listener
+ ) {
+ Map<String, ServerSocketChannel> socketsForNode = sockets.get(nodeId);
+ if (socketsForNode == null) {
+ return null;
+ }
+ ServerSocketChannel socket = socketsForNode.get(listener);
+ if (socket == null) {
+ return null;
+ }
+ usedSockets.computeIfAbsent(nodeId, __ -> new HashSet<>()).add(listener);
+ return socket;
+ }
+
+ /**
+ * Get or create a socket factory object associated with a given node ID.
+ *
+ * @param nodeId The ID of the node.
+ *
+ * @return The socket factory.
+ */
+ public synchronized PreboundSocketFactory getOrCreateSocketFactory(int nodeId) {
Review Comment:
> Could you please return the interface ServerSocketFactory instead? PreboundSocketFactory is a private class.
ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1735329187
##########
core/src/test/java/kafka/testkit/TestKitNode.java:
##########
@@ -35,6 +37,12 @@ default Set<String> logDataDirectories() {
return initialMetaPropertiesEnsemble().logDirProps().keySet();
}
+ default Uuid metadataUuid() {
Review Comment:
I will rename it to `metadataDirectoryId`, since that is more accurate than the current name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1735328645
##########
core/src/test/java/kafka/testkit/PreboundSocketFactoryManager.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.testkit;
+
+import kafka.server.ServerSocketFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.nio.channels.ServerSocketChannel;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+public class PreboundSocketFactoryManager implements AutoCloseable {
+ private static final Logger LOG = LoggerFactory.getLogger(PreboundSocketFactoryManager.class);
+
+ private class PreboundSocketFactory implements ServerSocketFactory {
+ private final int nodeId;
+
+ private PreboundSocketFactory(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ @Override
+ public ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException {
+ ServerSocketChannel socketChannel = getSocketForListenerAndMarkAsUsed(
+ nodeId,
+ listenerName);
+ if (socketChannel != null) {
+ return socketChannel;
+ }
+ return KafkaServerSocketFactory.INSTANCE.openServerSocket(
+ listenerName,
+ socketAddress,
+ listenBacklogSize,
+ recvBufferSize);
+ }
+ }
+
+ /**
+ * True if this manager is closed.
+ */
+ private boolean closed = false;
+
+ /**
+ * Maps node IDs to socket factory objects.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, PreboundSocketFactory> factories = new HashMap<>();
+
+ /**
+ * Maps node IDs to maps of listener names to ports.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, Map<String, ServerSocketChannel>> sockets = new HashMap<>();
+
+ /**
+ * Maps node IDs to set of the listeners that were used.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, Set<String>> usedSockets = new HashMap<>();
+
+ /**
+ * Get a socket from this manager, mark it as used, and return it.
+ *
+ * @param nodeId The ID of the node.
+ * @param listener The listener for the socket.
+ *
+ * @return null if the socket was not found; the socket, otherwise.
+ */
+ public synchronized ServerSocketChannel getSocketForListenerAndMarkAsUsed(
+ int nodeId,
+ String listener
+ ) {
+ Map<String, ServerSocketChannel> socketsForNode = sockets.get(nodeId);
+ if (socketsForNode == null) {
+ return null;
+ }
+ ServerSocketChannel socket = socketsForNode.get(listener);
+ if (socket == null) {
+ return null;
+ }
+ usedSockets.computeIfAbsent(nodeId, __ -> new HashSet<>()).add(listener);
+ return socket;
+ }
+
+ /**
+ * Get or create a socket factory object associated with a given node ID.
+ *
+ * @param nodeId The ID of the node.
+ *
+ * @return The socket factory.
+ */
+ public synchronized PreboundSocketFactory getOrCreateSocketFactory(int nodeId) {
+ return factories.computeIfAbsent(nodeId, __ -> new PreboundSocketFactory(nodeId));
+ }
+
+ /**
+ * Get a specific port number. The port will be created if it does not already exist.
+ *
+ * @param nodeId The ID of the node.
+ * @param listener The listener for the socket.
+ *
+ * @return The port number.
+ */
+ public synchronized int getOrCreatePortForListener(
+ int nodeId,
+ String listener
+ ) throws IOException {
+ Map<String, ServerSocketChannel> socketsForNode =
+ sockets.computeIfAbsent(nodeId, __ -> new HashMap<>());
+ ServerSocketChannel socketChannel = socketsForNode.get(listener);
+ if (socketChannel == null) {
+ if (closed) {
+ throw new RuntimeException("Cannot open new socket: manager is closed.");
+ }
+ socketChannel = ServerSocketFactory.KafkaServerSocketFactory.INSTANCE.openServerSocket(
+ listener,
+ new InetSocketAddress(0),
+ -1,
+ -1);
+ socketsForNode.put(listener, socketChannel);
+ }
+ InetSocketAddress socketAddress = (InetSocketAddress) socketChannel.getLocalAddress();
+ return socketAddress.getPort();
+ }
+
+ @Override
+ public synchronized void close() throws Exception {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ // Close all sockets that haven't been used by a SocketServer. (We don't want to close the
+ // ones that have been used by a SocketServer because that is the responsibility of that
+ // SocketServer.)
+ for (Entry<Integer, Map<String, ServerSocketChannel>> socketsEntry : sockets.entrySet()) {
+ Set<String> usedListeners = usedSockets.getOrDefault(
+ socketsEntry.getKey(), Collections.emptySet());
+ for (Entry<String, ServerSocketChannel> entry : socketsEntry.getValue().entrySet()) {
+ if (!usedListeners.contains(entry.getKey())) {
+ try {
Review Comment:
`ServerSocketChannel` doesn't implement `AutoCloseable`, unfortunately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1742439377
##########
core/src/test/java/kafka/testkit/KafkaClusterTestKit.java:
##########
@@ -404,22 +423,44 @@ private void formatNode(
boolean writeMetadataDirectory
) {
try {
- MetaPropertiesEnsemble.Copier copier =
- new MetaPropertiesEnsemble.Copier(MetaPropertiesEnsemble.EMPTY);
- for (Entry<String, MetaProperties> entry : ensemble.logDirProps().entrySet()) {
- String logDir = entry.getKey();
- if (writeMetadataDirectory || (!ensemble.metadataLogDir().equals(Optional.of(logDir)))) {
- log.trace("Adding {} to the list of directories to format.", logDir);
- copier.setLogDirProps(logDir, entry.getValue());
+ Formatter formatter = new Formatter();
+ formatter.setNodeId(ensemble.nodeId().getAsInt());
+ formatter.setClusterId(ensemble.clusterId().get());
+ if (writeMetadataDirectory) {
+ formatter.setDirectories(ensemble.logDirProps().keySet());
+ } else {
+ formatter.setDirectories(ensemble.logDirProps().keySet().stream().
+ filter(d -> !ensemble.metadataLogDir().get().equals(d)).
+ collect(Collectors.toSet()));
+ }
+ if (formatter.directories().isEmpty()) {
+ return;
+ }
+ formatter.setReleaseVersion(nodes.bootstrapMetadata().metadataVersion());
+ formatter.setFeatureLevel(KRaftVersion.FEATURE_NAME,
+ nodes.bootstrapMetadata().featureLevel(KRaftVersion.FEATURE_NAME));
+ formatter.setUnstableFeatureVersionsEnabled(true);
+ formatter.setIgnoreFormatted(false);
+ formatter.setControllerListenerName("CONTROLLER");
+ if (writeMetadataDirectory) {
+ formatter.setMetadataLogDirectory(ensemble.metadataLogDir().get());
+ } else {
+ formatter.setMetadataLogDirectory(Optional.empty());
Review Comment:
the default value of `metadataLogDirectory` is `Optional.empty()`, so `setMetadataLogDirectory(Optional.empty())` is no-op. However, that should be fine since it looks like more readable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1745897350
##########
core/src/test/java/kafka/test/junit/RaftClusterInvocationContext.java:
##########
@@ -247,52 +241,16 @@ public Map<Integer, ControllerServer> controllers() {
public void format() throws Exception {
if (formated.compareAndSet(false, true)) {
- Map<String, Features> nameToSupportedFeature = new TreeMap<>();
- Features.PRODUCTION_FEATURES.forEach(feature -> nameToSupportedFeature.put(feature.featureName(), feature));
- Map<String, Short> newFeatureLevels = new TreeMap<>();
-
- // Verify that all specified features are known to us.
- for (Map.Entry<Features, Short> entry : clusterConfig.features().entrySet()) {
- String featureName = entry.getKey().featureName();
- short level = entry.getValue();
- if (!featureName.equals(MetadataVersion.FEATURE_NAME)) {
- if (!nameToSupportedFeature.containsKey(featureName)) {
- throw new FormatterException("Unsupported feature: " + featureName +
- ". Supported features are: " + String.join(", ", nameToSupportedFeature.keySet()));
- }
- }
- newFeatureLevels.put(featureName, level);
- }
- newFeatureLevels.put(MetadataVersion.FEATURE_NAME, clusterConfig.metadataVersion().featureLevel());
-
- // Add default values for features that were not specified.
- Features.PRODUCTION_FEATURES.forEach(supportedFeature -> {
Review Comment:
I think the features should be spelled out in ClusterTest and also in our builds, for consistency with the current code.
Should be fixed now...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on PR #16991:
URL: https://github.com/apache/kafka/pull/16991#issuecomment-2458089844
We added this test in https://github.com/apache/kafka/pull/17582 instead. Closing this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "kevin-wu24 (via GitHub)" <gi...@apache.org>.
kevin-wu24 commented on PR #16991:
URL: https://github.com/apache/kafka/pull/16991#issuecomment-2435737601
@chia7712 I'm continuing Colin's work in this PR: https://github.com/apache/kafka/pull/17582. Can you take a look when you get the chance? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on PR #16991:
URL: https://github.com/apache/kafka/pull/16991#issuecomment-2316314527
> @cmccabe thanks for this patch.
>
> > In a follow-on PR, we should convert QuorumTestHarness to the new mechanism and remove the futures-based mechanism.
> Pardon me, have you opened the jira?
Thanks for the reminder. I filed it as https://issues.apache.org/jira/browse/KAFKA-17440
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1732488674
##########
core/src/test/java/kafka/testkit/TestKitNode.java:
##########
@@ -35,6 +37,12 @@ default Set<String> logDataDirectories() {
return initialMetaPropertiesEnsemble().logDirProps().keySet();
}
+ default Uuid metadataUuid() {
Review Comment:
It looks like `directoryId` rather than `metadataUuid`. I means `DynamicVoter` use the word `directoryId` :)
##########
core/src/test/java/kafka/testkit/PreboundSocketFactoryManager.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.testkit;
+
+import kafka.server.ServerSocketFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.nio.channels.ServerSocketChannel;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+public class PreboundSocketFactoryManager implements AutoCloseable {
+ private static final Logger LOG = LoggerFactory.getLogger(PreboundSocketFactoryManager.class);
+
+ private class PreboundSocketFactory implements ServerSocketFactory {
+ private final int nodeId;
+
+ private PreboundSocketFactory(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ @Override
+ public ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException {
+ ServerSocketChannel socketChannel = getSocketForListenerAndMarkAsUsed(
+ nodeId,
+ listenerName);
+ if (socketChannel != null) {
+ return socketChannel;
+ }
+ return KafkaServerSocketFactory.INSTANCE.openServerSocket(
+ listenerName,
+ socketAddress,
+ listenBacklogSize,
+ recvBufferSize);
+ }
+ }
+
+ /**
+ * True if this manager is closed.
+ */
+ private boolean closed = false;
+
+ /**
+ * Maps node IDs to socket factory objects.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, PreboundSocketFactory> factories = new HashMap<>();
+
+ /**
+ * Maps node IDs to maps of listener names to ports.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, Map<String, ServerSocketChannel>> sockets = new HashMap<>();
+
+ /**
+ * Maps node IDs to set of the listeners that were used.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, Set<String>> usedSockets = new HashMap<>();
+
+ /**
+ * Get a socket from this manager, mark it as used, and return it.
+ *
+ * @param nodeId The ID of the node.
+ * @param listener The listener for the socket.
+ *
+ * @return null if the socket was not found; the socket, otherwise.
+ */
+ public synchronized ServerSocketChannel getSocketForListenerAndMarkAsUsed(
+ int nodeId,
+ String listener
+ ) {
+ Map<String, ServerSocketChannel> socketsForNode = sockets.get(nodeId);
+ if (socketsForNode == null) {
+ return null;
+ }
+ ServerSocketChannel socket = socketsForNode.get(listener);
+ if (socket == null) {
+ return null;
+ }
+ usedSockets.computeIfAbsent(nodeId, __ -> new HashSet<>()).add(listener);
+ return socket;
+ }
+
+ /**
+ * Get or create a socket factory object associated with a given node ID.
+ *
+ * @param nodeId The ID of the node.
+ *
+ * @return The socket factory.
+ */
+ public synchronized PreboundSocketFactory getOrCreateSocketFactory(int nodeId) {
Review Comment:
Could you please return the interface `ServerSocketFactory` instead? `PreboundSocketFactory` is a private class.
Or we can use lambda instead. for example:
```java
return factories.computeIfAbsent(nodeId, __ -> (listenerName, socketAddress, listenBacklogSize, recvBufferSize) -> {
ServerSocketChannel socketChannel = getSocketForListenerAndMarkAsUsed(
nodeId,
listenerName);
if (socketChannel != null) {
return socketChannel;
}
return ServerSocketFactory.KafkaServerSocketFactory.INSTANCE.openServerSocket(
listenerName,
socketAddress,
listenBacklogSize,
recvBufferSize);
});
```
##########
core/src/test/java/kafka/testkit/PreboundSocketFactoryManager.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.testkit;
+
+import kafka.server.ServerSocketFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.nio.channels.ServerSocketChannel;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+public class PreboundSocketFactoryManager implements AutoCloseable {
+ private static final Logger LOG = LoggerFactory.getLogger(PreboundSocketFactoryManager.class);
+
+ private class PreboundSocketFactory implements ServerSocketFactory {
+ private final int nodeId;
+
+ private PreboundSocketFactory(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ @Override
+ public ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException {
+ ServerSocketChannel socketChannel = getSocketForListenerAndMarkAsUsed(
+ nodeId,
+ listenerName);
+ if (socketChannel != null) {
+ return socketChannel;
+ }
+ return KafkaServerSocketFactory.INSTANCE.openServerSocket(
+ listenerName,
+ socketAddress,
+ listenBacklogSize,
+ recvBufferSize);
+ }
+ }
+
+ /**
+ * True if this manager is closed.
+ */
+ private boolean closed = false;
+
+ /**
+ * Maps node IDs to socket factory objects.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, PreboundSocketFactory> factories = new HashMap<>();
+
+ /**
+ * Maps node IDs to maps of listener names to ports.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, Map<String, ServerSocketChannel>> sockets = new HashMap<>();
+
+ /**
+ * Maps node IDs to set of the listeners that were used.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, Set<String>> usedSockets = new HashMap<>();
+
+ /**
+ * Get a socket from this manager, mark it as used, and return it.
+ *
+ * @param nodeId The ID of the node.
+ * @param listener The listener for the socket.
+ *
+ * @return null if the socket was not found; the socket, otherwise.
+ */
+ public synchronized ServerSocketChannel getSocketForListenerAndMarkAsUsed(
+ int nodeId,
+ String listener
+ ) {
+ Map<String, ServerSocketChannel> socketsForNode = sockets.get(nodeId);
+ if (socketsForNode == null) {
+ return null;
+ }
+ ServerSocketChannel socket = socketsForNode.get(listener);
+ if (socket == null) {
+ return null;
+ }
+ usedSockets.computeIfAbsent(nodeId, __ -> new HashSet<>()).add(listener);
+ return socket;
+ }
+
+ /**
+ * Get or create a socket factory object associated with a given node ID.
+ *
+ * @param nodeId The ID of the node.
+ *
+ * @return The socket factory.
+ */
+ public synchronized PreboundSocketFactory getOrCreateSocketFactory(int nodeId) {
+ return factories.computeIfAbsent(nodeId, __ -> new PreboundSocketFactory(nodeId));
+ }
+
+ /**
+ * Get a specific port number. The port will be created if it does not already exist.
+ *
+ * @param nodeId The ID of the node.
+ * @param listener The listener for the socket.
+ *
+ * @return The port number.
+ */
+ public synchronized int getOrCreatePortForListener(
+ int nodeId,
+ String listener
+ ) throws IOException {
+ Map<String, ServerSocketChannel> socketsForNode =
+ sockets.computeIfAbsent(nodeId, __ -> new HashMap<>());
+ ServerSocketChannel socketChannel = socketsForNode.get(listener);
+ if (socketChannel == null) {
+ if (closed) {
+ throw new RuntimeException("Cannot open new socket: manager is closed.");
+ }
+ socketChannel = ServerSocketFactory.KafkaServerSocketFactory.INSTANCE.openServerSocket(
+ listener,
+ new InetSocketAddress(0),
+ -1,
+ -1);
+ socketsForNode.put(listener, socketChannel);
+ }
+ InetSocketAddress socketAddress = (InetSocketAddress) socketChannel.getLocalAddress();
+ return socketAddress.getPort();
+ }
+
+ @Override
+ public synchronized void close() throws Exception {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ // Close all sockets that haven't been used by a SocketServer. (We don't want to close the
+ // ones that have been used by a SocketServer because that is the responsibility of that
+ // SocketServer.)
+ for (Entry<Integer, Map<String, ServerSocketChannel>> socketsEntry : sockets.entrySet()) {
+ Set<String> usedListeners = usedSockets.getOrDefault(
+ socketsEntry.getKey(), Collections.emptySet());
+ for (Entry<String, ServerSocketChannel> entry : socketsEntry.getValue().entrySet()) {
+ if (!usedListeners.contains(entry.getKey())) {
+ try {
Review Comment:
How about using `Utils.closeQuietly`
##########
metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java:
##########
@@ -125,7 +125,7 @@ public class Formatter {
/**
* The metadata log directory.
*/
- private String metadataLogDirectory = null;
+ private Optional<String> metadataLogDirectory = null;
Review Comment:
Pardon me, why we can't have empty `metadataLogDirectory` (`Optional.empty`) by default?
##########
core/src/main/java/kafka/server/ServerSocketFactory.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.server;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.network.Selectable;
+import org.apache.kafka.common.utils.Utils;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.net.SocketException;
+import java.nio.channels.ServerSocketChannel;
+
+public interface ServerSocketFactory {
+ ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException;
+
+ class KafkaServerSocketFactory implements ServerSocketFactory {
+ public static final KafkaServerSocketFactory INSTANCE = new KafkaServerSocketFactory();
+
+ @Override
+ public ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException {
+ ServerSocketChannel socketChannel = null;
+ try {
+ socketChannel = ServerSocketChannel.open();
Review Comment:
Could you please rewrite it by following style to avoid resource leak.
```java
ServerSocketChannel socketChannel = ServerSocketChannel.open();
try {
socketChannel.configureBlocking(false);
```
##########
core/src/test/java/kafka/testkit/KafkaClusterTestKit.java:
##########
@@ -404,22 +423,44 @@ private void formatNode(
boolean writeMetadataDirectory
) {
try {
- MetaPropertiesEnsemble.Copier copier =
- new MetaPropertiesEnsemble.Copier(MetaPropertiesEnsemble.EMPTY);
- for (Entry<String, MetaProperties> entry : ensemble.logDirProps().entrySet()) {
- String logDir = entry.getKey();
- if (writeMetadataDirectory || (!ensemble.metadataLogDir().equals(Optional.of(logDir)))) {
- log.trace("Adding {} to the list of directories to format.", logDir);
- copier.setLogDirProps(logDir, entry.getValue());
+ Formatter formatter = new Formatter();
+ formatter.setNodeId(ensemble.nodeId().getAsInt());
+ formatter.setClusterId(ensemble.clusterId().get());
+ if (writeMetadataDirectory) {
+ formatter.setDirectories(ensemble.logDirProps().keySet());
+ } else {
+ formatter.setDirectories(ensemble.logDirProps().keySet().stream().
+ filter(d -> !ensemble.metadataLogDir().get().equals(d)).
+ collect(Collectors.toSet()));
+ }
+ if (formatter.directories().isEmpty()) {
+ return;
+ }
+ formatter.setReleaseVersion(nodes.bootstrapMetadata().metadataVersion());
+ formatter.setFeatureLevel(KRaftVersion.FEATURE_NAME,
+ nodes.bootstrapMetadata().featureLevel(KRaftVersion.FEATURE_NAME));
+ formatter.setUnstableFeatureVersionsEnabled(true);
+ formatter.setIgnoreFormatted(false);
+ formatter.setControllerListenerName("CONTROLLER");
+ if (writeMetadataDirectory) {
+ formatter.setMetadataLogDirectory(ensemble.metadataLogDir().get());
+ } else {
+ formatter.setMetadataLogDirectory(Optional.empty());
+ }
+ if (nodes.bootstrapMetadata().featureLevel(KRaftVersion.FEATURE_NAME) > 0) {
+ StringBuilder dynamicVotersBuilder = new StringBuilder();
+ String prefix = "";
+ for (TestKitNode controllerNode : nodes.controllerNodes().values()) {
+ int port = socketFactoryManager.
+ getOrCreatePortForListener(controllerNode.id(), "CONTROLLER");
+ dynamicVotersBuilder.append(prefix);
+ prefix = ",";
+ dynamicVotersBuilder.append(String.format("%d@localhost:%d:%s",
Review Comment:
Given that conflicts caused by base64 in converting UUID to String, maybe we can create `DynamicVoter` directly to avoid possible bugs in the future. for example:
```java
List<DynamicVoter> voters = new ArrayList<>();
for (TestKitNode controllerNode : nodes.controllerNodes().values()) {
int port = socketFactoryManager.
getOrCreatePortForListener(controllerNode.id(), "CONTROLLER");
voters.add(new DynamicVoter(controllerNode.metadataUuid(), controllerNode.id(), "localhost", port));
}
formatter.setInitialVoters(new DynamicVoters(voters));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1745897350
##########
core/src/test/java/kafka/test/junit/RaftClusterInvocationContext.java:
##########
@@ -247,52 +241,16 @@ public Map<Integer, ControllerServer> controllers() {
public void format() throws Exception {
if (formated.compareAndSet(false, true)) {
- Map<String, Features> nameToSupportedFeature = new TreeMap<>();
- Features.PRODUCTION_FEATURES.forEach(feature -> nameToSupportedFeature.put(feature.featureName(), feature));
- Map<String, Short> newFeatureLevels = new TreeMap<>();
-
- // Verify that all specified features are known to us.
- for (Map.Entry<Features, Short> entry : clusterConfig.features().entrySet()) {
- String featureName = entry.getKey().featureName();
- short level = entry.getValue();
- if (!featureName.equals(MetadataVersion.FEATURE_NAME)) {
- if (!nameToSupportedFeature.containsKey(featureName)) {
- throw new FormatterException("Unsupported feature: " + featureName +
- ". Supported features are: " + String.join(", ", nameToSupportedFeature.keySet()));
- }
- }
- newFeatureLevels.put(featureName, level);
- }
- newFeatureLevels.put(MetadataVersion.FEATURE_NAME, clusterConfig.metadataVersion().featureLevel());
-
- // Add default values for features that were not specified.
- Features.PRODUCTION_FEATURES.forEach(supportedFeature -> {
Review Comment:
I think the features should be spelled out in ClusterTest and also in our builds, for consistency with the current code.
Should be fixed now...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1745897350
##########
core/src/test/java/kafka/test/junit/RaftClusterInvocationContext.java:
##########
@@ -247,52 +241,16 @@ public Map<Integer, ControllerServer> controllers() {
public void format() throws Exception {
if (formated.compareAndSet(false, true)) {
- Map<String, Features> nameToSupportedFeature = new TreeMap<>();
- Features.PRODUCTION_FEATURES.forEach(feature -> nameToSupportedFeature.put(feature.featureName(), feature));
- Map<String, Short> newFeatureLevels = new TreeMap<>();
-
- // Verify that all specified features are known to us.
- for (Map.Entry<Features, Short> entry : clusterConfig.features().entrySet()) {
- String featureName = entry.getKey().featureName();
- short level = entry.getValue();
- if (!featureName.equals(MetadataVersion.FEATURE_NAME)) {
- if (!nameToSupportedFeature.containsKey(featureName)) {
- throw new FormatterException("Unsupported feature: " + featureName +
- ". Supported features are: " + String.join(", ", nameToSupportedFeature.keySet()));
- }
- }
- newFeatureLevels.put(featureName, level);
- }
- newFeatureLevels.put(MetadataVersion.FEATURE_NAME, clusterConfig.metadataVersion().featureLevel());
-
- // Add default values for features that were not specified.
- Features.PRODUCTION_FEATURES.forEach(supportedFeature -> {
Review Comment:
good point. I have a fix for this now... take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1735326142
##########
core/src/main/java/kafka/server/ServerSocketFactory.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.server;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.network.Selectable;
+import org.apache.kafka.common.utils.Utils;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.net.SocketException;
+import java.nio.channels.ServerSocketChannel;
+
+public interface ServerSocketFactory {
+ ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException;
+
+ class KafkaServerSocketFactory implements ServerSocketFactory {
+ public static final KafkaServerSocketFactory INSTANCE = new KafkaServerSocketFactory();
+
+ @Override
+ public ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException {
+ ServerSocketChannel socketChannel = null;
+ try {
+ socketChannel = ServerSocketChannel.open();
Review Comment:
Hmm... there shouldn't be a resource leak in the current case where it's initialized to null, since that's handled in the `catch` block. However, I can initialize the socketChannel variable with the result of `ServerSocketChannel.open`. That will make the code shorter and there's no disadvantage that I can see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #16991:
URL: https://github.com/apache/kafka/pull/16991#issuecomment-2318178248
@cmccabe could you please fix conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #16991:
URL: https://github.com/apache/kafka/pull/16991#discussion_r1738650234
##########
metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java:
##########
@@ -401,7 +409,7 @@ void doFormat(BootstrapMetadata bootstrapMetadata) throws Exception {
Map<String, DirectoryType> directoryTypes = new HashMap<>();
for (String emptyLogDir : ensemble.emptyLogDirs()) {
DirectoryType directoryType = DirectoryType.calculate(emptyLogDir,
- metadataLogDirectory,
+ metadataLogDirectory.orElseGet(() -> ""),
Review Comment:
`orElse("")` is good enough :smile:
##########
core/src/test/java/kafka/testkit/KafkaClusterTestKit.java:
##########
@@ -404,22 +423,44 @@ private void formatNode(
boolean writeMetadataDirectory
) {
try {
- MetaPropertiesEnsemble.Copier copier =
- new MetaPropertiesEnsemble.Copier(MetaPropertiesEnsemble.EMPTY);
- for (Entry<String, MetaProperties> entry : ensemble.logDirProps().entrySet()) {
- String logDir = entry.getKey();
- if (writeMetadataDirectory || (!ensemble.metadataLogDir().equals(Optional.of(logDir)))) {
- log.trace("Adding {} to the list of directories to format.", logDir);
- copier.setLogDirProps(logDir, entry.getValue());
+ Formatter formatter = new Formatter();
+ formatter.setNodeId(ensemble.nodeId().getAsInt());
+ formatter.setClusterId(ensemble.clusterId().get());
+ if (writeMetadataDirectory) {
+ formatter.setDirectories(ensemble.logDirProps().keySet());
+ } else {
+ formatter.setDirectories(ensemble.logDirProps().keySet().stream().
+ filter(d -> !ensemble.metadataLogDir().get().equals(d)).
+ collect(Collectors.toSet()));
+ }
+ if (formatter.directories().isEmpty()) {
+ return;
+ }
+ formatter.setReleaseVersion(nodes.bootstrapMetadata().metadataVersion());
+ formatter.setFeatureLevel(KRaftVersion.FEATURE_NAME,
+ nodes.bootstrapMetadata().featureLevel(KRaftVersion.FEATURE_NAME));
+ formatter.setUnstableFeatureVersionsEnabled(true);
+ formatter.setIgnoreFormatted(false);
+ formatter.setControllerListenerName("CONTROLLER");
+ if (writeMetadataDirectory) {
+ formatter.setMetadataLogDirectory(ensemble.metadataLogDir().get());
+ } else {
+ formatter.setMetadataLogDirectory(Optional.empty());
Review Comment:
the default value is empty, so maybe we can remove it.
##########
metadata/src/test/java/org/apache/kafka/metadata/bootstrap/BootstrapMetadataTest.java:
##########
@@ -71,10 +72,50 @@ public void testFromRecordsListWithoutMetadataVersion() {
() -> BootstrapMetadata.fromRecords(emptyList(), "quux")).getMessage());
}
+ private static final ApiMessageAndVersion MV_10 =
+ new ApiMessageAndVersion(new FeatureLevelRecord().
+ setName(FEATURE_NAME).
+ setFeatureLevel((short) 10), (short) 0);
+
+ private static final ApiMessageAndVersion MV_11 =
Review Comment:
This is unused. Maybe we should add a UT for it?
```java
assertEquals((short) 11, BootstrapMetadata.
fromRecords(Arrays.asList(MV_10, MV_11), "src").featureLevel(FEATURE_NAME));
```
##########
core/src/main/java/kafka/server/ServerSocketFactory.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.server;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.network.Selectable;
+import org.apache.kafka.common.utils.Utils;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.net.SocketException;
+import java.nio.channels.ServerSocketChannel;
+
+public interface ServerSocketFactory {
+ ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException;
+
+ class KafkaServerSocketFactory implements ServerSocketFactory {
+ public static final KafkaServerSocketFactory INSTANCE = new KafkaServerSocketFactory();
+
+ @Override
+ public ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException {
+ ServerSocketChannel socketChannel = ServerSocketChannel.open();
+ try {
+ socketChannel.configureBlocking(false);
+ if (recvBufferSize != Selectable.USE_DEFAULT_BUFFER_SIZE) {
+ socketChannel.socket().setReceiveBufferSize(recvBufferSize);
+ }
+ socketChannel.socket().bind(socketAddress, listenBacklogSize);
+ } catch (SocketException e) {
+ Utils.closeQuietly(socketChannel, "server socket");
+ throw new KafkaException(String.format("Socket server failed to bind to %s:%d: %s.",
Review Comment:
The `IOException` is added the method signature, so do we need to wrap the `SocketException` to `KafkaException`? maybe we can re-throw `SocketException` after releasing the `socketChannel`
##########
core/src/test/java/kafka/testkit/PreboundSocketFactoryManager.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.testkit;
+
+import kafka.server.ServerSocketFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.nio.channels.ServerSocketChannel;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+public class PreboundSocketFactoryManager implements AutoCloseable {
+ private static final Logger LOG = LoggerFactory.getLogger(PreboundSocketFactoryManager.class);
+
+ private class PreboundSocketFactory implements ServerSocketFactory {
+ private final int nodeId;
+
+ private PreboundSocketFactory(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ @Override
+ public ServerSocketChannel openServerSocket(
+ String listenerName,
+ InetSocketAddress socketAddress,
+ int listenBacklogSize,
+ int recvBufferSize
+ ) throws IOException {
+ ServerSocketChannel socketChannel = getSocketForListenerAndMarkAsUsed(
+ nodeId,
+ listenerName);
+ if (socketChannel != null) {
+ return socketChannel;
+ }
+ return KafkaServerSocketFactory.INSTANCE.openServerSocket(
+ listenerName,
+ socketAddress,
+ listenBacklogSize,
+ recvBufferSize);
+ }
+ }
+
+ /**
+ * True if this manager is closed.
+ */
+ private boolean closed = false;
+
+ /**
+ * Maps node IDs to socket factory objects.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, PreboundSocketFactory> factories = new HashMap<>();
+
+ /**
+ * Maps node IDs to maps of listener names to ports.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, Map<String, ServerSocketChannel>> sockets = new HashMap<>();
+
+ /**
+ * Maps node IDs to set of the listeners that were used.
+ * Protected by the object lock.
+ */
+ private final Map<Integer, Set<String>> usedSockets = new HashMap<>();
+
+ /**
+ * Get a socket from this manager, mark it as used, and return it.
+ *
+ * @param nodeId The ID of the node.
+ * @param listener The listener for the socket.
+ *
+ * @return null if the socket was not found; the socket, otherwise.
+ */
+ public synchronized ServerSocketChannel getSocketForListenerAndMarkAsUsed(
+ int nodeId,
+ String listener
+ ) {
+ Map<String, ServerSocketChannel> socketsForNode = sockets.get(nodeId);
+ if (socketsForNode == null) {
+ return null;
+ }
+ ServerSocketChannel socket = socketsForNode.get(listener);
+ if (socket == null) {
+ return null;
+ }
+ usedSockets.computeIfAbsent(nodeId, __ -> new HashSet<>()).add(listener);
+ return socket;
+ }
+
+ /**
+ * Get or create a socket factory object associated with a given node ID.
+ *
+ * @param nodeId The ID of the node.
+ *
+ * @return The socket factory.
+ */
+ public synchronized PreboundSocketFactory getOrCreateSocketFactory(int nodeId) {
+ return factories.computeIfAbsent(nodeId, __ -> new PreboundSocketFactory(nodeId));
+ }
+
+ /**
+ * Get a specific port number. The port will be created if it does not already exist.
+ *
+ * @param nodeId The ID of the node.
+ * @param listener The listener for the socket.
+ *
+ * @return The port number.
+ */
+ public synchronized int getOrCreatePortForListener(
+ int nodeId,
+ String listener
+ ) throws IOException {
+ Map<String, ServerSocketChannel> socketsForNode =
+ sockets.computeIfAbsent(nodeId, __ -> new HashMap<>());
+ ServerSocketChannel socketChannel = socketsForNode.get(listener);
+ if (socketChannel == null) {
+ if (closed) {
+ throw new RuntimeException("Cannot open new socket: manager is closed.");
+ }
+ socketChannel = ServerSocketFactory.KafkaServerSocketFactory.INSTANCE.openServerSocket(
+ listener,
+ new InetSocketAddress(0),
+ -1,
+ -1);
+ socketsForNode.put(listener, socketChannel);
+ }
+ InetSocketAddress socketAddress = (InetSocketAddress) socketChannel.getLocalAddress();
+ return socketAddress.getPort();
+ }
+
+ @Override
+ public synchronized void close() throws Exception {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ // Close all sockets that haven't been used by a SocketServer. (We don't want to close the
+ // ones that have been used by a SocketServer because that is the responsibility of that
+ // SocketServer.)
+ for (Entry<Integer, Map<String, ServerSocketChannel>> socketsEntry : sockets.entrySet()) {
+ Set<String> usedListeners = usedSockets.getOrDefault(
+ socketsEntry.getKey(), Collections.emptySet());
+ for (Entry<String, ServerSocketChannel> entry : socketsEntry.getValue().entrySet()) {
+ if (!usedListeners.contains(entry.getKey())) {
+ try {
Review Comment:
> ServerSocketChannel doesn't implement AutoCloseable, unfortunately.
not really. please take a look at https://docs.oracle.com/javase/8/docs/api/java/nio/channels/ServerSocketChannel.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #16991:
URL: https://github.com/apache/kafka/pull/16991#issuecomment-2336778660
@cmccabe Could you please fix build error?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: add ReconfigurableQuorumIntegrationTest [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe closed pull request #16991: MINOR: add ReconfigurableQuorumIntegrationTest
URL: https://github.com/apache/kafka/pull/16991
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org