You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2020/09/01 16:08:13 UTC
[GitHub] [ignite] timoninmaxim opened a new pull request #8206: IGNITE-13204 Thin client kubernetes auto discovery
timoninmaxim opened a new pull request #8206:
URL: https://github.com/apache/ignite/pull/8206
1. ReliableChannel now works with dynamic list of server addresses;
2. ClientConfiguration can be configured with addressFinder;
3. Provide Kubernetes addressFinder.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r501246454
##########
File path: modules/indexing/pom.xml
##########
@@ -133,6 +133,14 @@
<version>${guava.version}</version>
<scope>test</scope>
</dependency>
+
+ <dependency>
Review comment:
Absolutely, I got error if remove this dependency:
```
java.lang.NoClassDefFoundError: org/mockito/Mockito
at org.apache.ignite.internal.client.thin.ReliableChannelTest.checkFailAfterSendOperation(ReliableChannelTest.java:316)
at org.apache.ignite.internal.client.thin.ReliableChannelTest.testFailOnAsyncAfterSendOperation(ReliableChannelTest.java:273)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
Caused by: java.lang.ClassNotFoundException: org.mockito.Mockito
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 32 more
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r497490000
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -519,29 +689,44 @@ private boolean applyReconnectionThrottling() {
/**
* Get or create channel.
*/
- private synchronized ClientChannel getOrCreateChannel()
+ private ClientChannel getOrCreateChannel()
throws ClientConnectionException, ClientAuthenticationException, ClientProtocolError {
return getOrCreateChannel(false);
}
/**
* Get or create channel.
*/
- private synchronized ClientChannel getOrCreateChannel(boolean ignoreThrottling)
+ private ClientChannel getOrCreateChannel(boolean ignoreThrottling)
throws ClientConnectionException, ClientAuthenticationException, ClientProtocolError {
- if (ch == null) {
- if (!ignoreThrottling && applyReconnectionThrottling())
- throw new ClientConnectionException("Reconnect is not allowed due to applied throttling");
+ if (ch == null && !close) {
+ synchronized (this) {
+ if (close)
+ return null;
+
+ if (ch != null)
+ return ch;
- ch = chFactory.apply(chCfg);
+ if (!ignoreThrottling && applyReconnectionThrottling())
+ throw new ClientConnectionException("Reconnect is not allowed due to applied throttling");
- if (ch.serverNodeId() != null) {
- ch.addTopologyChangeListener(ReliableChannel.this::onTopologyChanged);
- ch.addNotificationListener(ReliableChannel.this);
+ ClientChannel channel = chFactory.apply(chCfg);
- nodeChannels.values().remove(this);
+ if (channel.serverNodeId() != null) {
+ channel.addTopologyChangeListener(ReliableChannel.this::onTopologyChanged);
+ channel.addNotificationListener(ReliableChannel.this);
- nodeChannels.putIfAbsent(ch.serverNodeId(), this);
+ if (serverNodeId == null)
+ serverNodeId = channel.serverNodeId();
+
+ if (serverNodeId != null && serverNodeId != channel.serverNodeId())
+ nodeChannels.remove(serverNodeId);
Review comment:
fixed.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -561,5 +746,24 @@ private synchronized void closeChannel() {
ch = null;
}
}
+
+ /** Close holder. */
+ void close() {
+ close = true;
+ if (serverNodeId != null)
+ nodeChannels.remove(serverNodeId);
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482847371
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(
+ new ClientChannelConfiguration(clientCfg, addr), chFactory, onChannelCreate, onChannelClose);
+
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
- affinityCtx = new ClientCacheAffinityContext(binary);
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
- ClientConnectionException lastEx = null;
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
- for (int i = 0; i < channels.length; i++) {
+ curChannelsGuard.writeLock().lock();
Review comment:
There are different goals for those sync operations:
1. Synchronized method guarantees that only one thread is initializing holders. This is required to limit requests to the AddressFinder, and avoid do the same work for prepare addresses multiple times;
2. AtomicReference is required as list of addresses are changed due to the preparation - merging new addresses, removing obsolete addresses. This performed in the "thin-client-channel#" thread.
3. The same time channels and CurChIdx are used in non-synchronized methods in user threads. There should be a guarantee for consistency between those variables. So this a reason to use the guard.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r483465905
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/spi/discovery/tcp/ipfinder/kubernetes/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes;
+
+import java.net.InetAddress;
+import java.util.function.Supplier;
+import org.apache.ignite.kubernetes.KubernetesConnectorConfigurator;
+import org.apache.ignite.kubernetes.KubernetesConnectorDefaults;
+import org.apache.ignite.kubernetes.KubernetesServiceAddressResolver;
+
+/**
+ * Address finder for automatic lookup of Ignite nodes running in Kubernetes environment. All Ignite nodes have to
+ * deployed as Kubernetes pods in order to be found. Applications and Ignite nodes running outside of Kubernetes
+ * will not be able to reach the containerized counterparts.
+ * <p>
+ * The implementation is based on a distinct Kubernetes service that has to be created and should be deployed prior
+ * Ignite nodes startup. The service will maintain a list of all endpoints (internal IP addresses) of all containerized
+ * Ignite pods running so far. The name of the service must be equal to {@link #setServiceName(String)} which is
+ * `ignite` by default.
+ * <p>
+ * As for Ignite pods, it's recommended to label them in such a way that the service will use the label in its selector
Review comment:
What you describe is correct, but also obvious - you have to use labels to define a service with a subset of pods in k8s. This is common knowledge and this is the first thing k8s docs tell you. So:
1. This part is redundant
2. This part is worded in such a way that (in my opinion) can bring unnecessary confusion.
Anyway, this is a minor thing - keep it if you insist.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r484050231
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/spi/discovery/tcp/ipfinder/kubernetes/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes;
+
+import java.net.InetAddress;
+import java.util.function.Supplier;
+import org.apache.ignite.kubernetes.KubernetesConnectorConfigurator;
+import org.apache.ignite.kubernetes.KubernetesConnectorDefaults;
+import org.apache.ignite.kubernetes.KubernetesServiceAddressResolver;
+
+/**
+ * Address finder for automatic lookup of Ignite nodes running in Kubernetes environment. All Ignite nodes have to
+ * deployed as Kubernetes pods in order to be found. Applications and Ignite nodes running outside of Kubernetes
+ * will not be able to reach the containerized counterparts.
+ * <p>
+ * The implementation is based on a distinct Kubernetes service that has to be created and should be deployed prior
+ * Ignite nodes startup. The service will maintain a list of all endpoints (internal IP addresses) of all containerized
+ * Ignite pods running so far. The name of the service must be equal to {@link #setServiceName(String)} which is
+ * `ignite` by default.
+ * <p>
+ * As for Ignite pods, it's recommended to label them in such a way that the service will use the label in its selector
Review comment:
I think you're correct. I reword it, just add mention about nodes types that are target by service. I think this mention is important as invalid configuration would lead to performance issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r487016500
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
`initCollection` perform async operations, so it will lead to escaping incomplete ReliableChannel. I'm not confident is there too much troubles in our particular case, but best practice is to avoid such situations. IMHO, lazy initialization is preferable but there are too much test cases that rely that connection is initialized from beginning. So let's do not break the API, but preserve this object safe.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
`initCollection` perform async operations, so it will lead to escaping incomplete ReliableChannel. I'm not confident is there too much troubles in our particular case, but best practice is to avoid such situations. IMHO, lazy initialization is preferable but there are too much test cases that rely that connection is initialized from beginning. So let's do not break the API, but preserve this object safe.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
`initCollection` perform async operations, so it will lead to escaping incomplete ReliableChannel. I'm not confident is there too much troubles in our particular case, but best practice is to avoid such situations. IMHO, lazy initialization is preferable but there are too much test cases that rely that connection is initialized from beginning. So let's do not break the API, but preserve this object safe.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
`initCollection` perform async operations, so it will lead to escaping incomplete ReliableChannel. I'm not confident is there too much troubles in our particular case, but best practice is to avoid such situations. IMHO, lazy initialization is preferable but there are too much test cases that rely that connection is initialized from beginning. So let's do not break the API, but preserve this object safe.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
`initCollection` perform async operations, so it will lead to escaping incomplete ReliableChannel. I'm not confident is there too much troubles in our particular case, but best practice is to avoid such situations. IMHO, lazy initialization is preferable but there are too much test cases that rely that connection is initialized from beginning. So let's do not break the API, but preserve this object safe.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
`initCollection` perform async operations, so it will lead to escaping incomplete ReliableChannel. I'm not confident is there too much troubles in our particular case, but best practice is to avoid such situations. IMHO, lazy initialization is preferable but there are too much test cases that rely that connection is initialized from beginning. So let's do not break the API, but preserve this object safe.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
`initCollection` perform async operations, so it will lead to escaping incomplete ReliableChannel. I'm not confident is there too much troubles in our particular case, but best practice is to avoid such situations. IMHO, lazy initialization is preferable but there are too much test cases that rely that connection is initialized from beginning. So let's do not break the API, but preserve this object safe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r497290792
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** */
+ private void checkDoesNotReinit(ClientConfiguration ccfg) {
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ List<ReliableChannel.ClientChannelHolder> originalChannels = rc.getChannelHolders();
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+
+ assertSame(originalChannels, newChannels);
+ IntStream.range(0, 3).forEach(i -> {
+ assertSame(originalChannels.get(i), newChannels.get(i));
+ assertFalse(originalChannels.get(i).isClosed());
+ });
+ assertEquals(3, newChannels.size());
+ }
+
+ /** Checks that node channels are persisted if channels are reinit with static address configuration. */
+ @Test
+ public void testNodeChannelsAreNotCleaned() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ // Trigger TestClientChannel creation.
+ rc.service(null, null, null);
+
+ assertEquals(1, rc.nodeChannels.size());
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+
+ assertEquals(1, rc.nodeChannels.size());
+ }
+
+ /** Checks that channels are changed (add new, remove old) and close channels if reinitialization performed. */
+ @Test
+ public void testDynamicAddressReinitializedCorrectly() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8003"));
+
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+
+ List<ReliableChannel.ClientChannelHolder> originChannels = Collections.unmodifiableList(rc.getChannelHolders());
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+
+ List<ReliableChannel.ClientChannelHolder> closedChannels = originChannels.stream()
+ .filter(r -> r.isClosed())
+ .collect(Collectors.toList());
+
+ assertEquals(2, closedChannels.size());
+
+ List<ReliableChannel.ClientChannelHolder> reuseChannel = originChannels.stream()
+ .filter(c -> !c.isClosed())
+ .collect(Collectors.toList());
+
+ assertEquals(1, reuseChannel.size());
+
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+ assertEquals(2, newChannels.size());
+ assertTrue(newChannels.get(0) == reuseChannel.get(0) || newChannels.get(1) == reuseChannel.get(0));
Review comment:
`equals` for `ClientChannelHolder` only compare references :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r488674198
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +126,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
-
- affinityCtx = new ClientCacheAffinityContext(binary);
-
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
+ affCtx = new ClientCacheAffinityContext(binary);
+ }
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
+ applyOnDefaultChannel(channel -> {
+ // do nothing, just trigger channel connection.
Review comment:
Fixed.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -356,7 +318,7 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
/**
* @return host:port_range address lines parsed as {@link InetSocketAddress}.
*/
- private static List<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
+ private Set<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
Review comment:
Fixed.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
Review comment:
Fixed.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
Review comment:
FIxed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r487716600
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
Review comment:
At least you can skip these steps if references to addresses are equals (in case of k8s you will get the new array each time, in case of static configuration - the same array each time)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r501070383
##########
File path: modules/indexing/pom.xml
##########
@@ -133,6 +133,14 @@
<version>${guava.version}</version>
<scope>test</scope>
</dependency>
+
+ <dependency>
Review comment:
Unfortunately it's required. By some reason, `ClientTestSuite` lives in the indexing module while some tests (including `ReliableChannelTest` that is a part of the suite) are in the core module. So all dependencies that are required for tests in the core must be set within the indexing module too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r501070383
##########
File path: modules/indexing/pom.xml
##########
@@ -133,6 +133,14 @@
<version>${guava.version}</version>
<scope>test</scope>
</dependency>
+
+ <dependency>
Review comment:
Unfortunately it's required. By some reason, `ClientTestSuit` lives in indexing module while some tests (including `ReliableChannelTest` that is a part of the suite) are in core module. So all dependencies that requires for tests in core must be set within indexing module too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504779913
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -512,80 +504,81 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
return ranges.stream()
.flatMap(r -> IntStream
.rangeClosed(r.portFrom(), r.portTo()).boxed()
- .map(p -> new InetSocketAddress(r.host(), p))
+ .map(p -> InetSocketAddress.createUnresolved(r.host(), p))
)
- .collect(Collectors.toList());
+ .collect(Collectors.toMap(a -> a, a -> 1, Integer::sum));
}
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
+ /**
+ * Roll current default channel if specified holder equals to it.
+ */
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
+ int idx = curChIdx;
+ List<ClientChannelHolder> holders = channels;
- throw e;
- }
- }
+ ClientChannelHolder dfltHld = holders.get(idx);
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ if (dfltHld == hld) {
+ idx += 1;
+
+ if (idx >= holders.size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On current channel failure.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
+ private void onChannelFailure(ClientChannel ch) {
// There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
// when current index was changed and no other wrong channel will be closed by current thread because
// onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ onChannelFailure(channels.get(curChIdx), ch);
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ chFailLsnrs.forEach(Runnable::run);
+
+ if (scheduledChannelsReinit.get())
+ channelsInit(true);
+ else
+ rollCurrentChannel(hld);
Review comment:
if channel fails and there is scheduled reinit, there are chances that reinitialization will fix the issue (reinitialization may assign new default channel).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r487016500
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
`initCollection` perform async operations, so it will lead to escaping incomplete ReliableChannel. I'm not confident is there too much troubles in our particular case, but best practice is to avoid such situations. IMHO, lazy initialization is preferable but there are too much test cases that rely that connection is initialized from beginning. So let's do not break the API, but preserve this object safe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r487686584
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
Review comment:
Somebody can configure addresses with DNS names, in k8s env it is possible with StatefulSet configuration. But node can change IP. So after topology change we should check resolve all addresses. Nevertheless, It's possible to cache result of resolution.
In case of static configuration holders are persisted. But it's true that cache help to win some by skip iteration steps.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r487016500
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
`initCollection` perform async operations, so it will lead to escaping incomplete ReliableChannel. I'm not confident is there too much troubles in our particular case, but best practice is to avoid such situations. IMHO, lazy initialization is preferable but there are too much test cases that rely that connection is initialized from beginning. So let's do not break the API, but preserve this object safe.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
`initCollection` perform async operations, so it will lead to escaping incomplete ReliableChannel. I'm not confident is there too much troubles in our particular case, but best practice is to avoid such situations. IMHO, lazy initialization is preferable but there are too much test cases that rely that connection is initialized from beginning. So let's do not break the API, but preserve this object safe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r488665705
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
Review comment:
fixed.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
Review comment:
fixed.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
+ this.addrs = Arrays.copyOf(addrs, addrs.length);
+ addrFinder = () -> this.addrs;
+ }
+
+ return this;
+ }
+
+ /**
+ * @param finder function that finds node addresses
+ */
+ public ClientConfiguration setAddressesFinder(Supplier<String[]> finder) {
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r494267952
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +135,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
-
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
affinityCtx = new ClientCacheAffinityContext(binary);
+ }
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
-
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
Review comment:
FIxed, add test for that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r486971241
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
Review comment:
Same here - let's revert and move the logic to `ReliableChannel`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
+ this.addrs = Arrays.copyOf(addrs, addrs.length);
+ addrFinder = () -> this.addrs;
+ }
+
+ return this;
+ }
+
+ /**
+ * @param finder function that finds node addresses
+ */
+ public ClientConfiguration setAddressesFinder(Supplier<String[]> finder) {
Review comment:
Corresponding getter is missing.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -169,8 +167,10 @@
// No-op.
}
- for (ClientChannelHolder hld : channels)
- hld.closeChannel();
+ if (channels.get() != null) {
+ for (ClientChannelHolder hld: channels.get())
Review comment:
Here and below: race condition - `channels` may become null after the null check. Copy to a variable, then check for null / iterate / etc.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
/** Index of the current channel. */
- private int curChIdx;
+ private volatile int curChIdx = -1;
/** Partition awareness enabled. */
private final boolean partitionAwarenessEnabled;
/** Cache partition awareness context. */
- private final ClientCacheAffinityContext affinityCtx;
+ private final ClientCacheAffinityContext affCtx;
Review comment:
Please revert unnecessary renames to reduce the scope of changes.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
Is there a reason to bring this outside of the constructor?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
Review comment:
`AtomicReference` is not needed, we can just use `volatile` since only `get()` and `set()` methods are used below.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -493,6 +520,25 @@ public ClientConfiguration setReconnectThrottlingRetries(int reconnectThrottling
return this;
}
+ /**
+ * Get retry limit.
+ */
+ public int getRetryLimit() {
+ return retryLimit;
+ }
+
+ /**
+ * Try use limited number of channels to send a request if default channel is not responding.
Review comment:
```
Sets the retry limit. When a request fails due to a connection error, and multiple server connections are available, Ignite will retry the request on every connection. When this property is greater than zero, Ignite will limit the number of retries.
```
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
Review comment:
Let's revert - no need to be clever here. `ClientConfiguration` should be a simple POJO, the logic for address handling should not be here.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -381,78 +343,59 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
.rangeClosed(r.portFrom(), r.portTo()).boxed()
.map(p -> new InetSocketAddress(r.host(), p))
)
- .collect(Collectors.toList());
- }
-
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
- try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
-
- throw e;
- }
- }
-
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ .collect(Collectors.toSet());
}
/**
- * On current channel failure.
+ * Roll current default channel if specified holder equals to it.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
- // There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
- // when current index was changed and no other wrong channel will be closed by current thread because
- // onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
+ try {
+ ClientChannelHolder dfltHld = channels.get().get(curChIdx);
+ if (dfltHld == hld) {
+ int idx = curChIdx + 1;
+ if (idx >= channels.get().size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (hld != null && ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ rollCurrentChannel(hld);
+
+ chFailLsnrs.forEach(Runnable::run);
}
/**
* Asynchronously try to establish a connection to all configured servers.
*/
- private void initAllChannelsAsync() {
- // Skip if there is already channels reinit scheduled.
- if (scheduledChannelsReinit.compareAndSet(false, true)) {
- asyncRunner.submit(
- () -> {
- scheduledChannelsReinit.set(false);
-
- for (ClientChannelHolder hld : channels) {
- if (scheduledChannelsReinit.get() || closed)
- return; // New reinit task scheduled or channel is closed.
-
- try {
- hld.getOrCreateChannel(true);
- }
- catch (Exception ignore) {
- // No-op.
- }
+ void initAllChannelsAsync() {
Review comment:
`private`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
Review comment:
`shouldStopChannelsReinit()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
Review comment:
This requires iteration over entire map. Instead, use `addrs.entrySet()` for looping and remove by key.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
+ if (!force && channels.get() != null)
+ return;
+
+ // Skip if there is already channels reinit scheduled.
+ // Flag is set back when a thread comes in synchronized initChannelHolders
+ if (scheduledChannelsReinit.compareAndSet(false, true)) {
+ initChannelHolders(force);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = Optional
Review comment:
Too clever, verbose, and memory-hungry. Just make it `channel = hld == null ? null : hld.getOrCreateChannel();`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -33,9 +36,16 @@
/** Serial version uid. */
private static final long serialVersionUID = 0L;
- /** @serial Server addresses. */
+ /**
+ * @serial Server addresses.
+ * @deprecated deprecated in favor of {@link #addrFinder}. Keep it due to Serializable compatibility.
+ */
+ @Deprecated
private String[] addrs = null;
+ /** Server addresses finder. */
+ private transient Supplier<String[]> addrFinder;
Review comment:
I would prefer a dedicated interface - `ThinClientAddressFinder` or something like this.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
Review comment:
Same here - let's revert and move the logic to `ReliableChannel`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
+ this.addrs = Arrays.copyOf(addrs, addrs.length);
+ addrFinder = () -> this.addrs;
+ }
+
+ return this;
+ }
+
+ /**
+ * @param finder function that finds node addresses
+ */
+ public ClientConfiguration setAddressesFinder(Supplier<String[]> finder) {
Review comment:
Corresponding getter is missing.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -169,8 +167,10 @@
// No-op.
}
- for (ClientChannelHolder hld : channels)
- hld.closeChannel();
+ if (channels.get() != null) {
+ for (ClientChannelHolder hld: channels.get())
Review comment:
Here and below: race condition - `channels` may become null after the null check. Copy to a variable, then check for null / iterate / etc.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
/** Index of the current channel. */
- private int curChIdx;
+ private volatile int curChIdx = -1;
/** Partition awareness enabled. */
private final boolean partitionAwarenessEnabled;
/** Cache partition awareness context. */
- private final ClientCacheAffinityContext affinityCtx;
+ private final ClientCacheAffinityContext affCtx;
Review comment:
Please revert unnecessary renames to reduce the scope of changes.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
Is there a reason to bring this outside of the constructor?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
Review comment:
`AtomicReference` is not needed, we can just use `volatile` since only `get()` and `set()` methods are used below.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -493,6 +520,25 @@ public ClientConfiguration setReconnectThrottlingRetries(int reconnectThrottling
return this;
}
+ /**
+ * Get retry limit.
+ */
+ public int getRetryLimit() {
+ return retryLimit;
+ }
+
+ /**
+ * Try use limited number of channels to send a request if default channel is not responding.
Review comment:
```
Sets the retry limit. When a request fails due to a connection error, and multiple server connections are available, Ignite will retry the request on every connection. When this property is greater than zero, Ignite will limit the number of retries.
```
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
Review comment:
Let's revert - no need to be clever here. `ClientConfiguration` should be a simple POJO, the logic for address handling should not be here.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -381,78 +343,59 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
.rangeClosed(r.portFrom(), r.portTo()).boxed()
.map(p -> new InetSocketAddress(r.host(), p))
)
- .collect(Collectors.toList());
- }
-
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
- try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
-
- throw e;
- }
- }
-
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ .collect(Collectors.toSet());
}
/**
- * On current channel failure.
+ * Roll current default channel if specified holder equals to it.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
- // There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
- // when current index was changed and no other wrong channel will be closed by current thread because
- // onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
+ try {
+ ClientChannelHolder dfltHld = channels.get().get(curChIdx);
+ if (dfltHld == hld) {
+ int idx = curChIdx + 1;
+ if (idx >= channels.get().size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (hld != null && ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ rollCurrentChannel(hld);
+
+ chFailLsnrs.forEach(Runnable::run);
}
/**
* Asynchronously try to establish a connection to all configured servers.
*/
- private void initAllChannelsAsync() {
- // Skip if there is already channels reinit scheduled.
- if (scheduledChannelsReinit.compareAndSet(false, true)) {
- asyncRunner.submit(
- () -> {
- scheduledChannelsReinit.set(false);
-
- for (ClientChannelHolder hld : channels) {
- if (scheduledChannelsReinit.get() || closed)
- return; // New reinit task scheduled or channel is closed.
-
- try {
- hld.getOrCreateChannel(true);
- }
- catch (Exception ignore) {
- // No-op.
- }
+ void initAllChannelsAsync() {
Review comment:
`private`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
Review comment:
`shouldStopChannelsReinit()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
Review comment:
This requires iteration over entire map. Instead, use `addrs.entrySet()` for looping and remove by key.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
+ if (!force && channels.get() != null)
+ return;
+
+ // Skip if there is already channels reinit scheduled.
+ // Flag is set back when a thread comes in synchronized initChannelHolders
+ if (scheduledChannelsReinit.compareAndSet(false, true)) {
+ initChannelHolders(force);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = Optional
Review comment:
Too clever, verbose, and memory-hungry. Just make it `channel = hld == null ? null : hld.getOrCreateChannel();`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -33,9 +36,16 @@
/** Serial version uid. */
private static final long serialVersionUID = 0L;
- /** @serial Server addresses. */
+ /**
+ * @serial Server addresses.
+ * @deprecated deprecated in favor of {@link #addrFinder}. Keep it due to Serializable compatibility.
+ */
+ @Deprecated
private String[] addrs = null;
+ /** Server addresses finder. */
+ private transient Supplier<String[]> addrFinder;
Review comment:
I would prefer a dedicated interface - `ThinClientAddressFinder` or something like this.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
Review comment:
Same here - let's revert and move the logic to `ReliableChannel`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
+ this.addrs = Arrays.copyOf(addrs, addrs.length);
+ addrFinder = () -> this.addrs;
+ }
+
+ return this;
+ }
+
+ /**
+ * @param finder function that finds node addresses
+ */
+ public ClientConfiguration setAddressesFinder(Supplier<String[]> finder) {
Review comment:
Corresponding getter is missing.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -169,8 +167,10 @@
// No-op.
}
- for (ClientChannelHolder hld : channels)
- hld.closeChannel();
+ if (channels.get() != null) {
+ for (ClientChannelHolder hld: channels.get())
Review comment:
Here and below: race condition - `channels` may become null after the null check. Copy to a variable, then check for null / iterate / etc.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
/** Index of the current channel. */
- private int curChIdx;
+ private volatile int curChIdx = -1;
/** Partition awareness enabled. */
private final boolean partitionAwarenessEnabled;
/** Cache partition awareness context. */
- private final ClientCacheAffinityContext affinityCtx;
+ private final ClientCacheAffinityContext affCtx;
Review comment:
Please revert unnecessary renames to reduce the scope of changes.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
Is there a reason to bring this outside of the constructor?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
Review comment:
`AtomicReference` is not needed, we can just use `volatile` since only `get()` and `set()` methods are used below.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -493,6 +520,25 @@ public ClientConfiguration setReconnectThrottlingRetries(int reconnectThrottling
return this;
}
+ /**
+ * Get retry limit.
+ */
+ public int getRetryLimit() {
+ return retryLimit;
+ }
+
+ /**
+ * Try use limited number of channels to send a request if default channel is not responding.
Review comment:
```
Sets the retry limit. When a request fails due to a connection error, and multiple server connections are available, Ignite will retry the request on every connection. When this property is greater than zero, Ignite will limit the number of retries.
```
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
Review comment:
Let's revert - no need to be clever here. `ClientConfiguration` should be a simple POJO, the logic for address handling should not be here.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -381,78 +343,59 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
.rangeClosed(r.portFrom(), r.portTo()).boxed()
.map(p -> new InetSocketAddress(r.host(), p))
)
- .collect(Collectors.toList());
- }
-
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
- try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
-
- throw e;
- }
- }
-
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ .collect(Collectors.toSet());
}
/**
- * On current channel failure.
+ * Roll current default channel if specified holder equals to it.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
- // There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
- // when current index was changed and no other wrong channel will be closed by current thread because
- // onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
+ try {
+ ClientChannelHolder dfltHld = channels.get().get(curChIdx);
+ if (dfltHld == hld) {
+ int idx = curChIdx + 1;
+ if (idx >= channels.get().size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (hld != null && ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ rollCurrentChannel(hld);
+
+ chFailLsnrs.forEach(Runnable::run);
}
/**
* Asynchronously try to establish a connection to all configured servers.
*/
- private void initAllChannelsAsync() {
- // Skip if there is already channels reinit scheduled.
- if (scheduledChannelsReinit.compareAndSet(false, true)) {
- asyncRunner.submit(
- () -> {
- scheduledChannelsReinit.set(false);
-
- for (ClientChannelHolder hld : channels) {
- if (scheduledChannelsReinit.get() || closed)
- return; // New reinit task scheduled or channel is closed.
-
- try {
- hld.getOrCreateChannel(true);
- }
- catch (Exception ignore) {
- // No-op.
- }
+ void initAllChannelsAsync() {
Review comment:
`private`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
Review comment:
`shouldStopChannelsReinit()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
Review comment:
This requires iteration over entire map. Instead, use `addrs.entrySet()` for looping and remove by key.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
+ if (!force && channels.get() != null)
+ return;
+
+ // Skip if there is already channels reinit scheduled.
+ // Flag is set back when a thread comes in synchronized initChannelHolders
+ if (scheduledChannelsReinit.compareAndSet(false, true)) {
+ initChannelHolders(force);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = Optional
Review comment:
Too clever, verbose, and memory-hungry. Just make it `channel = hld == null ? null : hld.getOrCreateChannel();`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -33,9 +36,16 @@
/** Serial version uid. */
private static final long serialVersionUID = 0L;
- /** @serial Server addresses. */
+ /**
+ * @serial Server addresses.
+ * @deprecated deprecated in favor of {@link #addrFinder}. Keep it due to Serializable compatibility.
+ */
+ @Deprecated
private String[] addrs = null;
+ /** Server addresses finder. */
+ private transient Supplier<String[]> addrFinder;
Review comment:
I would prefer a dedicated interface - `ThinClientAddressFinder` or something like this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn merged pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn merged pull request #8206:
URL: https://github.com/apache/ignite/pull/8206
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r494267419
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
+ */
+ @Test
+ public void testClientDiscoveryNodesJoin() throws Exception {
+ for (int i = 0; i < MAX_CLUSTER_SIZE; ++i) {
+ startGrid(i);
+ awaitPartitionMapExchange();
+
+ int[] workChannels = IntStream.rangeClosed(0, i).toArray();
+
+ if (i == 0)
+ initClient(getClientConfigurationWithDiscovery(), workChannels);
+ else {
+ detectTopologyChange();
+ awaitChannelsInit(workChannels);
+ }
+
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Test that client use channels to all running nodes while nodes stop
+ */
+ @Test
+ public void testClientDiscoveryNodesLeave() throws Exception {
+ startGrids(MAX_CLUSTER_SIZE);
+ awaitPartitionMapExchange();
+
+ initClient(getClientConfigurationWithDiscovery(), 0, 1, 2, 3);
+ detectTopologyChange();
+
+ for (int i = MAX_CLUSTER_SIZE - 1; i != 0; i--) {
+ int[] workChannels = IntStream.range(0, i).toArray();
+
+ channels[i] = null;
+ stopGrid(i);
+ awaitPartitionMapExchange();
+ detectTopologyChange();
+
+ awaitChannelsInit(workChannels);
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Checks that each request goes to right node.
+ */
+ private void testPartitionAwareness(int... chIdxs) {
+ ClientCache<Object, Object> clientCache = client.cache(PART_CACHE_NAME);
+ IgniteInternalCache<Object, Object> igniteCache = grid(0).context().cache().cache(PART_CACHE_NAME);
+
+ Map<TestTcpClientChannel, Boolean> channelHits = Arrays.stream(chIdxs).boxed()
+ .collect(Collectors.toMap(idx -> channels[idx], idx -> false));
+
+ for (int i = 0; i < KEY_CNT; i++) {
+ TestTcpClientChannel opCh = affinityChannel(i, igniteCache);
+
+ clientCache.put(i, i);
+
+ if (i == 0)
+ assertOpOnChannel(dfltCh, ClientOperation.CACHE_PARTITIONS);
+
+ assertOpOnChannel(opCh, ClientOperation.CACHE_PUT);
+ assertTrue(channelHits.containsKey(opCh));
+
+ channelHits.compute(opCh, (c, old) -> true);
+ }
+
+ assertFalse(channelHits.containsValue(false));
+ }
+
+ /**
+ * Provide ClientConfiguration with addrResolver that find all alive nodes
+ */
+ private ClientConfiguration getClientConfigurationWithDiscovery() {
+ ClientAddressFinder addrFinder = () ->
+ IgnitionEx.allGrids().stream().map(node -> {
+ int port = (Integer) node.cluster().localNode().attributes().get(CLIENT_LISTENER_PORT);
+ return "127.0.0.1:" + port;
+ }).toArray(String[]::new);
+
+ return new ClientConfiguration()
+ .setAddressesFinder(addrFinder)
+ .setPartitionAwarenessEnabled(true);
+ }
+
+ /**
+ * Trigger client to detect topology change
+ */
+ private void detectTopologyChange() {
+ // Send non-affinity request to detect topology change.
+ initDefaultChannel();
+ client.getOrCreateCache(PART_CACHE_NAME);
Review comment:
FIxed, add test for that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r486971241
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
Review comment:
Same here - let's revert and move the logic to `ReliableChannel`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
+ this.addrs = Arrays.copyOf(addrs, addrs.length);
+ addrFinder = () -> this.addrs;
+ }
+
+ return this;
+ }
+
+ /**
+ * @param finder function that finds node addresses
+ */
+ public ClientConfiguration setAddressesFinder(Supplier<String[]> finder) {
Review comment:
Corresponding getter is missing.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -169,8 +167,10 @@
// No-op.
}
- for (ClientChannelHolder hld : channels)
- hld.closeChannel();
+ if (channels.get() != null) {
+ for (ClientChannelHolder hld: channels.get())
Review comment:
Here and below: race condition - `channels` may become null after the null check. Copy to a variable, then check for null / iterate / etc.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
/** Index of the current channel. */
- private int curChIdx;
+ private volatile int curChIdx = -1;
/** Partition awareness enabled. */
private final boolean partitionAwarenessEnabled;
/** Cache partition awareness context. */
- private final ClientCacheAffinityContext affinityCtx;
+ private final ClientCacheAffinityContext affCtx;
Review comment:
Please revert unnecessary renames to reduce the scope of changes.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
Is there a reason to bring this outside of the constructor?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
Review comment:
`AtomicReference` is not needed, we can just use `volatile` since only `get()` and `set()` methods are used below.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -493,6 +520,25 @@ public ClientConfiguration setReconnectThrottlingRetries(int reconnectThrottling
return this;
}
+ /**
+ * Get retry limit.
+ */
+ public int getRetryLimit() {
+ return retryLimit;
+ }
+
+ /**
+ * Try use limited number of channels to send a request if default channel is not responding.
Review comment:
```
Sets the retry limit. When a request fails due to a connection error, and multiple server connections are available, Ignite will retry the request on every connection. When this property is greater than zero, Ignite will limit the number of retries.
```
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
Review comment:
Let's revert - no need to be clever here. `ClientConfiguration` should be a simple POJO, the logic for address handling should not be here.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -381,78 +343,59 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
.rangeClosed(r.portFrom(), r.portTo()).boxed()
.map(p -> new InetSocketAddress(r.host(), p))
)
- .collect(Collectors.toList());
- }
-
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
- try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
-
- throw e;
- }
- }
-
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ .collect(Collectors.toSet());
}
/**
- * On current channel failure.
+ * Roll current default channel if specified holder equals to it.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
- // There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
- // when current index was changed and no other wrong channel will be closed by current thread because
- // onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
+ try {
+ ClientChannelHolder dfltHld = channels.get().get(curChIdx);
+ if (dfltHld == hld) {
+ int idx = curChIdx + 1;
+ if (idx >= channels.get().size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (hld != null && ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ rollCurrentChannel(hld);
+
+ chFailLsnrs.forEach(Runnable::run);
}
/**
* Asynchronously try to establish a connection to all configured servers.
*/
- private void initAllChannelsAsync() {
- // Skip if there is already channels reinit scheduled.
- if (scheduledChannelsReinit.compareAndSet(false, true)) {
- asyncRunner.submit(
- () -> {
- scheduledChannelsReinit.set(false);
-
- for (ClientChannelHolder hld : channels) {
- if (scheduledChannelsReinit.get() || closed)
- return; // New reinit task scheduled or channel is closed.
-
- try {
- hld.getOrCreateChannel(true);
- }
- catch (Exception ignore) {
- // No-op.
- }
+ void initAllChannelsAsync() {
Review comment:
`private`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
Review comment:
`shouldStopChannelsReinit()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
Review comment:
This requires iteration over entire map. Instead, use `addrs.entrySet()` for looping and remove by key.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
+ if (!force && channels.get() != null)
+ return;
+
+ // Skip if there is already channels reinit scheduled.
+ // Flag is set back when a thread comes in synchronized initChannelHolders
+ if (scheduledChannelsReinit.compareAndSet(false, true)) {
+ initChannelHolders(force);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = Optional
Review comment:
Too clever, verbose, and memory-hungry. Just make it `channel = hld == null ? null : hld.getOrCreateChannel();`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482828960
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(
+ new ClientChannelConfiguration(clientCfg, addr), chFactory, onChannelCreate, onChannelClose);
+
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
- affinityCtx = new ClientCacheAffinityContext(binary);
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
- ClientConnectionException lastEx = null;
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
- for (int i = 0; i < channels.length; i++) {
+ curChannelsGuard.writeLock().lock();
Review comment:
Then maybe `AtomicReference` is not required? Furthermore, this method is `synchronized` - overall rather hard to understand.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504966233
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
Review comment:
I tried rewrite this code, but it only becomes more complicated (lambdas starts to complain about non-final scope, etc.). So, I suggest another approach with introducing a callback to store attempts counter in an external value (push commit). WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482984061
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannelFacade.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.ignite.IgniteBinary;
+import org.apache.ignite.client.ClientAuthenticationException;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+
+/**
+ * Communication channel with failover and partition awareness.
+ */
+final class ReliableChannelFacade implements AutoCloseable {
Review comment:
sure, I will remove this change from the PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504924954
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -512,80 +504,81 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
return ranges.stream()
.flatMap(r -> IntStream
.rangeClosed(r.portFrom(), r.portTo()).boxed()
- .map(p -> new InetSocketAddress(r.host(), p))
+ .map(p -> InetSocketAddress.createUnresolved(r.host(), p))
)
- .collect(Collectors.toList());
+ .collect(Collectors.toMap(a -> a, a -> 1, Integer::sum));
}
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
+ /**
+ * Roll current default channel if specified holder equals to it.
+ */
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
+ int idx = curChIdx;
+ List<ClientChannelHolder> holders = channels;
- throw e;
- }
- }
+ ClientChannelHolder dfltHld = holders.get(idx);
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ if (dfltHld == hld) {
+ idx += 1;
+
+ if (idx >= holders.size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On current channel failure.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
+ private void onChannelFailure(ClientChannel ch) {
// There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
// when current index was changed and no other wrong channel will be closed by current thread because
// onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ onChannelFailure(channels.get(curChIdx), ch);
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ chFailLsnrs.forEach(Runnable::run);
+
+ if (scheduledChannelsReinit.get())
+ channelsInit(true);
+ else
+ rollCurrentChannel(hld);
Review comment:
Actually there is an invokation sequence for default case: `channelsInit -> applyOnDefaultChannel` to establish connection with default channel (that checks that connection is OK). So rolling is not required too. Also rolling at least acquires lock so it is not free.
So, there is a matrix of cases for channel failure:
1. Fixed addreses, topology changes -> Channel is rolled with `channelsInit -> applyOnDefaultChannel`;
2. Fixed addresses, topology does not change -> `rollCurrentChannel`;
3. Dynamic address, topology changes -> channels are reinited twice, rolls with `applyOnDefaultChannel`;
4. Dynamic address, topology does not change -> `rollCurrentChannel`.
So, there is an issue with double initialization for dynamic addresses only. I will fix it by skipping second initialization as describe in other comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482832744
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/kubernetes/KubernetesConnectorConfigurator.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.kubernetes;
+
+/**
+ * Interface provide methods to configure Kubernetes connection.
+ */
+public interface KubernetesConnectorConfigurator {
Review comment:
I can move this interface to package with finders and make it package-private, isn't it enough? Public API for IpFinder isn't changed, it still have the same setters. I'm agree that this interface doesn't look good but it's a limitation of exisiting IpFinder. Alternatives are:
1. Remove this interface and track k8s configuration params only with constructor of `KubernetesServiceAddressResolver`;
2. Provide a new object like `KubernetesConnectionKonfiguration` instead of interface, provide new IpFinder constructor that accept the configuration, deprecate setters in the IpFinder in favor of the constructor.
I like the second option. But it lead change of the IpFinder API (add new, deprecate old), is it ok?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r497289186
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** */
+ private void checkDoesNotReinit(ClientConfiguration ccfg) {
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ List<ReliableChannel.ClientChannelHolder> originalChannels = rc.getChannelHolders();
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+
+ assertSame(originalChannels, newChannels);
+ IntStream.range(0, 3).forEach(i -> {
+ assertSame(originalChannels.get(i), newChannels.get(i));
Review comment:
But you compare current reference with current reference. If you want to check that reference for every holder doesn't change you should copy `originalChannels` before topology change and then check this copy against `newChannels`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504826278
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -512,80 +504,81 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
return ranges.stream()
.flatMap(r -> IntStream
.rangeClosed(r.portFrom(), r.portTo()).boxed()
- .map(p -> new InetSocketAddress(r.host(), p))
+ .map(p -> InetSocketAddress.createUnresolved(r.host(), p))
)
- .collect(Collectors.toList());
+ .collect(Collectors.toMap(a -> a, a -> 1, Integer::sum));
}
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
+ /**
+ * Roll current default channel if specified holder equals to it.
+ */
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
+ int idx = curChIdx;
+ List<ClientChannelHolder> holders = channels;
- throw e;
- }
- }
+ ClientChannelHolder dfltHld = holders.get(idx);
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ if (dfltHld == hld) {
+ idx += 1;
+
+ if (idx >= holders.size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On current channel failure.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
+ private void onChannelFailure(ClientChannel ch) {
// There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
// when current index was changed and no other wrong channel will be closed by current thread because
// onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ onChannelFailure(channels.get(curChIdx), ch);
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ chFailLsnrs.forEach(Runnable::run);
+
+ if (scheduledChannelsReinit.get())
+ channelsInit(true);
+ else
+ rollCurrentChannel(hld);
Review comment:
There are chances if some dynamic address finder is used, but by default, you just check current address twice and skip one channel. In both cases (dynamic and static address finder) there is nothing bad happens if we roll current channel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r494250150
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
+ */
+ @Test
+ public void testClientDiscoveryNodesJoin() throws Exception {
+ for (int i = 0; i < MAX_CLUSTER_SIZE; ++i) {
+ startGrid(i);
+ awaitPartitionMapExchange();
+
+ int[] workChannels = IntStream.rangeClosed(0, i).toArray();
+
+ if (i == 0)
+ initClient(getClientConfigurationWithDiscovery(), workChannels);
+ else {
+ detectTopologyChange();
+ awaitChannelsInit(workChannels);
+ }
+
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Test that client use channels to all running nodes while nodes stop
+ */
+ @Test
+ public void testClientDiscoveryNodesLeave() throws Exception {
+ startGrids(MAX_CLUSTER_SIZE);
+ awaitPartitionMapExchange();
+
+ initClient(getClientConfigurationWithDiscovery(), 0, 1, 2, 3);
+ detectTopologyChange();
+
+ for (int i = MAX_CLUSTER_SIZE - 1; i != 0; i--) {
+ int[] workChannels = IntStream.range(0, i).toArray();
+
+ channels[i] = null;
+ stopGrid(i);
+ awaitPartitionMapExchange();
+ detectTopologyChange();
+
+ awaitChannelsInit(workChannels);
Review comment:
fixied
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
+ */
+ @Test
+ public void testClientDiscoveryNodesJoin() throws Exception {
+ for (int i = 0; i < MAX_CLUSTER_SIZE; ++i) {
+ startGrid(i);
+ awaitPartitionMapExchange();
+
+ int[] workChannels = IntStream.rangeClosed(0, i).toArray();
+
+ if (i == 0)
+ initClient(getClientConfigurationWithDiscovery(), workChannels);
+ else {
+ detectTopologyChange();
+ awaitChannelsInit(workChannels);
+ }
+
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Test that client use channels to all running nodes while nodes stop
+ */
+ @Test
+ public void testClientDiscoveryNodesLeave() throws Exception {
Review comment:
Yes, it's ok that test will pass with default configuration too. It's like regression test, that discovery work expectedly. The case you describe will work the same way, it will pass both for default and discovery configurations, as both of them will configure the same set of channels. But it is a good test case and I've added it too.
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** */
+ private void checkDoesNotReinit(ClientConfiguration ccfg) {
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ List<ReliableChannel.ClientChannelHolder> originalChannels = rc.getChannelHolders();
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+
+ assertSame(originalChannels, newChannels);
+ IntStream.range(0, 3).forEach(i -> {
+ assertSame(originalChannels.get(i), newChannels.get(i));
Review comment:
FIxed, add test for that.
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
+ */
+ @Test
+ public void testClientDiscoveryNodesJoin() throws Exception {
+ for (int i = 0; i < MAX_CLUSTER_SIZE; ++i) {
+ startGrid(i);
+ awaitPartitionMapExchange();
+
+ int[] workChannels = IntStream.rangeClosed(0, i).toArray();
+
+ if (i == 0)
+ initClient(getClientConfigurationWithDiscovery(), workChannels);
+ else {
+ detectTopologyChange();
+ awaitChannelsInit(workChannels);
+ }
+
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Test that client use channels to all running nodes while nodes stop
+ */
+ @Test
+ public void testClientDiscoveryNodesLeave() throws Exception {
+ startGrids(MAX_CLUSTER_SIZE);
+ awaitPartitionMapExchange();
+
+ initClient(getClientConfigurationWithDiscovery(), 0, 1, 2, 3);
+ detectTopologyChange();
+
+ for (int i = MAX_CLUSTER_SIZE - 1; i != 0; i--) {
+ int[] workChannels = IntStream.range(0, i).toArray();
+
+ channels[i] = null;
+ stopGrid(i);
+ awaitPartitionMapExchange();
+ detectTopologyChange();
+
+ awaitChannelsInit(workChannels);
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Checks that each request goes to right node.
+ */
+ private void testPartitionAwareness(int... chIdxs) {
+ ClientCache<Object, Object> clientCache = client.cache(PART_CACHE_NAME);
+ IgniteInternalCache<Object, Object> igniteCache = grid(0).context().cache().cache(PART_CACHE_NAME);
+
+ Map<TestTcpClientChannel, Boolean> channelHits = Arrays.stream(chIdxs).boxed()
+ .collect(Collectors.toMap(idx -> channels[idx], idx -> false));
+
+ for (int i = 0; i < KEY_CNT; i++) {
+ TestTcpClientChannel opCh = affinityChannel(i, igniteCache);
+
+ clientCache.put(i, i);
+
+ if (i == 0)
+ assertOpOnChannel(dfltCh, ClientOperation.CACHE_PARTITIONS);
+
+ assertOpOnChannel(opCh, ClientOperation.CACHE_PUT);
+ assertTrue(channelHits.containsKey(opCh));
+
+ channelHits.compute(opCh, (c, old) -> true);
+ }
+
+ assertFalse(channelHits.containsValue(false));
+ }
+
+ /**
+ * Provide ClientConfiguration with addrResolver that find all alive nodes
+ */
+ private ClientConfiguration getClientConfigurationWithDiscovery() {
+ ClientAddressFinder addrFinder = () ->
+ IgnitionEx.allGrids().stream().map(node -> {
+ int port = (Integer) node.cluster().localNode().attributes().get(CLIENT_LISTENER_PORT);
+ return "127.0.0.1:" + port;
+ }).toArray(String[]::new);
+
+ return new ClientConfiguration()
+ .setAddressesFinder(addrFinder)
+ .setPartitionAwarenessEnabled(true);
+ }
+
+ /**
+ * Trigger client to detect topology change
+ */
+ private void detectTopologyChange() {
+ // Send non-affinity request to detect topology change.
+ initDefaultChannel();
+ client.getOrCreateCache(PART_CACHE_NAME);
Review comment:
Sorry, you're correct. Fixed.
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
+ */
+ @Test
+ public void testClientDiscoveryNodesJoin() throws Exception {
+ for (int i = 0; i < MAX_CLUSTER_SIZE; ++i) {
+ startGrid(i);
+ awaitPartitionMapExchange();
+
+ int[] workChannels = IntStream.rangeClosed(0, i).toArray();
+
+ if (i == 0)
+ initClient(getClientConfigurationWithDiscovery(), workChannels);
+ else {
+ detectTopologyChange();
+ awaitChannelsInit(workChannels);
+ }
+
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Test that client use channels to all running nodes while nodes stop
+ */
+ @Test
+ public void testClientDiscoveryNodesLeave() throws Exception {
+ startGrids(MAX_CLUSTER_SIZE);
+ awaitPartitionMapExchange();
+
+ initClient(getClientConfigurationWithDiscovery(), 0, 1, 2, 3);
+ detectTopologyChange();
+
+ for (int i = MAX_CLUSTER_SIZE - 1; i != 0; i--) {
+ int[] workChannels = IntStream.range(0, i).toArray();
+
+ channels[i] = null;
+ stopGrid(i);
+ awaitPartitionMapExchange();
+ detectTopologyChange();
+
+ awaitChannelsInit(workChannels);
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Checks that each request goes to right node.
+ */
+ private void testPartitionAwareness(int... chIdxs) {
+ ClientCache<Object, Object> clientCache = client.cache(PART_CACHE_NAME);
+ IgniteInternalCache<Object, Object> igniteCache = grid(0).context().cache().cache(PART_CACHE_NAME);
+
+ Map<TestTcpClientChannel, Boolean> channelHits = Arrays.stream(chIdxs).boxed()
+ .collect(Collectors.toMap(idx -> channels[idx], idx -> false));
+
+ for (int i = 0; i < KEY_CNT; i++) {
+ TestTcpClientChannel opCh = affinityChannel(i, igniteCache);
+
+ clientCache.put(i, i);
+
+ if (i == 0)
+ assertOpOnChannel(dfltCh, ClientOperation.CACHE_PARTITIONS);
+
+ assertOpOnChannel(opCh, ClientOperation.CACHE_PUT);
+ assertTrue(channelHits.containsKey(opCh));
+
+ channelHits.compute(opCh, (c, old) -> true);
+ }
+
+ assertFalse(channelHits.containsValue(false));
+ }
+
+ /**
+ * Provide ClientConfiguration with addrResolver that find all alive nodes
+ */
+ private ClientConfiguration getClientConfigurationWithDiscovery() {
+ ClientAddressFinder addrFinder = () ->
+ IgnitionEx.allGrids().stream().map(node -> {
+ int port = (Integer) node.cluster().localNode().attributes().get(CLIENT_LISTENER_PORT);
+ return "127.0.0.1:" + port;
+ }).toArray(String[]::new);
+
+ return new ClientConfiguration()
+ .setAddressesFinder(addrFinder)
+ .setPartitionAwarenessEnabled(true);
+ }
+
+ /**
+ * Trigger client to detect topology change
+ */
+ private void detectTopologyChange() {
+ // Send non-affinity request to detect topology change.
+ initDefaultChannel();
+ client.getOrCreateCache(PART_CACHE_NAME);
Review comment:
FIxed, add test for that
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +135,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
-
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
affinityCtx = new ClientCacheAffinityContext(binary);
+ }
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
-
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
Review comment:
FIxed, add test for that
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -461,9 +418,8 @@ private void initAllChannelsAsync() {
* @param ch Channel.
*/
private void onTopologyChanged(ClientChannel ch) {
- if (partitionAwarenessEnabled && affinityCtx.updateLastTopologyVersion(ch.serverTopologyVersion(),
- ch.serverNodeId()))
- initAllChannelsAsync();
+ if (affinityCtx.updateLastTopologyVersion(ch.serverTopologyVersion(), ch.serverNodeId()))
+ channelsInit(true);
Review comment:
Fair. Fixed it. Change signature of `initChannelHolders`, now it's not void but returns boolean (flag of interruption). Otherwise for disabled awareness and continuous topology events there would be a chances for StackOverflow, as channel failure trigger recursively `applyOnDefaultChannel` to enable default channel.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r488675311
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -33,9 +36,16 @@
/** Serial version uid. */
private static final long serialVersionUID = 0L;
- /** @serial Server addresses. */
+ /**
+ * @serial Server addresses.
+ * @deprecated deprecated in favor of {@link #addrFinder}. Keep it due to Serializable compatibility.
+ */
+ @Deprecated
private String[] addrs = null;
+ /** Server addresses finder. */
+ private transient Supplier<String[]> addrFinder;
Review comment:
Used `ClientAddressFinder` as all classes in this package use Client prefix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r483148749
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/spi/discovery/tcp/ipfinder/kubernetes/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes;
Review comment:
moved.
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/spi/discovery/tcp/ipfinder/kubernetes/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes;
+
+import java.net.InetAddress;
+import java.util.function.Supplier;
+import org.apache.ignite.kubernetes.KubernetesConnectorConfigurator;
+import org.apache.ignite.kubernetes.KubernetesConnectorDefaults;
+import org.apache.ignite.kubernetes.KubernetesServiceAddressResolver;
+
+/**
+ * Address finder for automatic lookup of Ignite nodes running in Kubernetes environment. All Ignite nodes have to
Review comment:
fixed.
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/spi/discovery/tcp/ipfinder/kubernetes/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes;
+
+import java.net.InetAddress;
+import java.util.function.Supplier;
+import org.apache.ignite.kubernetes.KubernetesConnectorConfigurator;
+import org.apache.ignite.kubernetes.KubernetesConnectorDefaults;
+import org.apache.ignite.kubernetes.KubernetesServiceAddressResolver;
+
+/**
+ * Address finder for automatic lookup of Ignite nodes running in Kubernetes environment. All Ignite nodes have to
+ * deployed as Kubernetes pods in order to be found. Applications and Ignite nodes running outside of Kubernetes
+ * will not be able to reach the containerized counterparts.
+ * <p>
+ * The implementation is based on a distinct Kubernetes service that has to be created and should be deployed prior
+ * Ignite nodes startup. The service will maintain a list of all endpoints (internal IP addresses) of all containerized
Review comment:
1. Agree with first comment, fixed;
2. Disagree, as Service do maintain Endpoints, from the link you provide:
> The controller for the Service selector continuously scans for Pods that match its selector, and then POSTs any updates to an Endpoint object
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482778585
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/kubernetes/KubernetesConnectorConfigurator.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.kubernetes;
+
+/**
+ * Interface provide methods to configure Kubernetes connection.
+ */
+public interface KubernetesConnectorConfigurator {
Review comment:
@ptupitsyn Connection to k8s API is independent on Ignite node type. IpFinder and AddressFinder both have the same setters to configure the connection (namespace, service, etc). IMHO the right implementation would be to create a class like `KubernetesConnectionConfiguration` and use it as field in the finders. But it will break existing API for IpFinder. So I introduced the interface to track common configuration params.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482855961
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannelFacade.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.ignite.IgniteBinary;
+import org.apache.ignite.client.ClientAuthenticationException;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+
+/**
+ * Communication channel with failover and partition awareness.
+ */
+final class ReliableChannelFacade implements AutoCloseable {
Review comment:
I think that `ReliableChannel` was overloaded, it was responsible for 2 things the same time: communication and affinity awareness. Due to that there were some non-intuitive conditions for me while I debug the code. So I split the original class on 2 parts:
1. `ReliableChannel` now is responsible for sending **any** request to Ignite nodes. If some channel is failure it tries to use another one, roll channels, etc. It just provides infrastructure for communication between client and servers.
2. `ReliableChannelFacade` provides logic with affinity awareness: send **affinity** requests, choose affinity node for a request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r501070383
##########
File path: modules/indexing/pom.xml
##########
@@ -133,6 +133,14 @@
<version>${guava.version}</version>
<scope>test</scope>
</dependency>
+
+ <dependency>
Review comment:
Unfortunately it's required. By some reason, `ClientTestSuit` lives in indexing module while some tests (including ` `ReliableChannelTest`) are in core module. So all dependencies that requires for tests in core must be set within indexing module too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r501208152
##########
File path: modules/core/src/main/java/org/apache/ignite/client/ClientAddressFinder.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.client;
+
+/**
+ * This interface provides a list of addresses of Ignite server nodes within a cluster. Thin client uses the list to
+ * route user requests. There are cases when the list is not static, for example in cloud environment. In such cases
+ * addresses of nodes and/or number of server nodes can change. Implementation of this interface should handle these.
+ *
+ * Ignite waites for a topology change to trigger {@link #getAddresses()}. There are two modes of how soon Ignite calls
Review comment:
`waites` -> `waits`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r486971241
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
Review comment:
Same here - let's revert and move the logic to `ReliableChannel`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
+ this.addrs = Arrays.copyOf(addrs, addrs.length);
+ addrFinder = () -> this.addrs;
+ }
+
+ return this;
+ }
+
+ /**
+ * @param finder function that finds node addresses
+ */
+ public ClientConfiguration setAddressesFinder(Supplier<String[]> finder) {
Review comment:
Corresponding getter is missing.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -169,8 +167,10 @@
// No-op.
}
- for (ClientChannelHolder hld : channels)
- hld.closeChannel();
+ if (channels.get() != null) {
+ for (ClientChannelHolder hld: channels.get())
Review comment:
Here and below: race condition - `channels` may become null after the null check. Copy to a variable, then check for null / iterate / etc.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
/** Index of the current channel. */
- private int curChIdx;
+ private volatile int curChIdx = -1;
/** Partition awareness enabled. */
private final boolean partitionAwarenessEnabled;
/** Cache partition awareness context. */
- private final ClientCacheAffinityContext affinityCtx;
+ private final ClientCacheAffinityContext affCtx;
Review comment:
Please revert unnecessary renames to reduce the scope of changes.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
Is there a reason to bring this outside of the constructor?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
Review comment:
`AtomicReference` is not needed, we can just use `volatile` since only `get()` and `set()` methods are used below.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -493,6 +520,25 @@ public ClientConfiguration setReconnectThrottlingRetries(int reconnectThrottling
return this;
}
+ /**
+ * Get retry limit.
+ */
+ public int getRetryLimit() {
+ return retryLimit;
+ }
+
+ /**
+ * Try use limited number of channels to send a request if default channel is not responding.
Review comment:
```
Sets the retry limit. When a request fails due to a connection error, and multiple server connections are available, Ignite will retry the request on every connection. When this property is greater than zero, Ignite will limit the number of retries.
```
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
Review comment:
Let's revert - no need to be clever here. `ClientConfiguration` should be a simple POJO, the logic for address handling should not be here.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -381,78 +343,59 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
.rangeClosed(r.portFrom(), r.portTo()).boxed()
.map(p -> new InetSocketAddress(r.host(), p))
)
- .collect(Collectors.toList());
- }
-
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
- try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
-
- throw e;
- }
- }
-
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ .collect(Collectors.toSet());
}
/**
- * On current channel failure.
+ * Roll current default channel if specified holder equals to it.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
- // There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
- // when current index was changed and no other wrong channel will be closed by current thread because
- // onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
+ try {
+ ClientChannelHolder dfltHld = channels.get().get(curChIdx);
+ if (dfltHld == hld) {
+ int idx = curChIdx + 1;
+ if (idx >= channels.get().size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (hld != null && ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ rollCurrentChannel(hld);
+
+ chFailLsnrs.forEach(Runnable::run);
}
/**
* Asynchronously try to establish a connection to all configured servers.
*/
- private void initAllChannelsAsync() {
- // Skip if there is already channels reinit scheduled.
- if (scheduledChannelsReinit.compareAndSet(false, true)) {
- asyncRunner.submit(
- () -> {
- scheduledChannelsReinit.set(false);
-
- for (ClientChannelHolder hld : channels) {
- if (scheduledChannelsReinit.get() || closed)
- return; // New reinit task scheduled or channel is closed.
-
- try {
- hld.getOrCreateChannel(true);
- }
- catch (Exception ignore) {
- // No-op.
- }
+ void initAllChannelsAsync() {
Review comment:
`private`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
Review comment:
`shouldStopChannelsReinit()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
Review comment:
This requires iteration over entire map. Instead, use `addrs.entrySet()` for looping and remove by key.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
+ if (!force && channels.get() != null)
+ return;
+
+ // Skip if there is already channels reinit scheduled.
+ // Flag is set back when a thread comes in synchronized initChannelHolders
+ if (scheduledChannelsReinit.compareAndSet(false, true)) {
+ initChannelHolders(force);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = Optional
Review comment:
Too clever, verbose, and memory-hungry. Just make it `channel = hld == null ? null : hld.getOrCreateChannel();`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -33,9 +36,16 @@
/** Serial version uid. */
private static final long serialVersionUID = 0L;
- /** @serial Server addresses. */
+ /**
+ * @serial Server addresses.
+ * @deprecated deprecated in favor of {@link #addrFinder}. Keep it due to Serializable compatibility.
+ */
+ @Deprecated
private String[] addrs = null;
+ /** Server addresses finder. */
+ private transient Supplier<String[]> addrFinder;
Review comment:
I would prefer a dedicated interface - `ThinClientAddressFinder` or something like this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482769793
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +116,33 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /**
+ * Try use other limited number of channels to send a request if default channel is not responding.
+ * 0 means try use all configured channels before fail.
+ */
+ private int channelsAttemptsLimit = 0;
Review comment:
1. Do we really need this property? What is the use case when this is not `0`?
2. I think `retryLimit` is a better name
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482858155
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannelFacade.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.ignite.IgniteBinary;
+import org.apache.ignite.client.ClientAuthenticationException;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+
+/**
+ * Communication channel with failover and partition awareness.
+ */
+final class ReliableChannelFacade implements AutoCloseable {
Review comment:
I'd like to ask you to limit the scope of changes in this PR, it is hard to grasp as it is. Can we revert all unrelated refactoring and do it separately later?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504774568
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
Review comment:
Currently master branch do work approach you describe, and it leads to:
1. introducing additional AtomicInteger variable for attempts counter. Counter is required as it used in separate threads execute `handleServiceAsync` method;
2. code duplication for iterating through channels (for sync and async methods).
This is reasons to use T2. It's not pretty, but the alternative isn't looks better, I think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r494251704
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
+ */
+ @Test
+ public void testClientDiscoveryNodesJoin() throws Exception {
+ for (int i = 0; i < MAX_CLUSTER_SIZE; ++i) {
+ startGrid(i);
+ awaitPartitionMapExchange();
+
+ int[] workChannels = IntStream.rangeClosed(0, i).toArray();
+
+ if (i == 0)
+ initClient(getClientConfigurationWithDiscovery(), workChannels);
+ else {
+ detectTopologyChange();
+ awaitChannelsInit(workChannels);
+ }
+
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Test that client use channels to all running nodes while nodes stop
+ */
+ @Test
+ public void testClientDiscoveryNodesLeave() throws Exception {
Review comment:
Yes, it's ok that test will pass with default configuration too. It's like regression test, that discovery work expectedly. The case you describe will work the same way, it will pass both for default and discovery configurations, as both of them will configure the same set of channels. But it is a good test case and I've added it too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482983089
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/spi/discovery/tcp/ipfinder/kubernetes/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes;
+
+import java.net.InetAddress;
+import java.util.function.Supplier;
+import org.apache.ignite.kubernetes.KubernetesConnectorConfigurator;
+import org.apache.ignite.kubernetes.KubernetesConnectorDefaults;
+import org.apache.ignite.kubernetes.KubernetesServiceAddressResolver;
+
+/**
+ * Address finder for automatic lookup of Ignite nodes running in Kubernetes environment. All Ignite nodes have to
+ * deployed as Kubernetes pods in order to be found. Applications and Ignite nodes running outside of Kubernetes
+ * will not be able to reach the containerized counterparts.
+ * <p>
+ * The implementation is based on a distinct Kubernetes service that has to be created and should be deployed prior
+ * Ignite nodes startup. The service will maintain a list of all endpoints (internal IP addresses) of all containerized
+ * Ignite pods running so far. The name of the service must be equal to {@link #setServiceName(String)} which is
+ * `ignite` by default.
+ * <p>
+ * As for Ignite pods, it's recommended to label them in such a way that the service will use the label in its selector
Review comment:
Actually, we should care. K8S services are bounded with k8s deployments by selector. IpFinder / AddressFinder return addresses to all pods that are matches service. So, for IpFinder service should contain all server and thick clients nodes. For AddressFinder service should contain only server nodes. Otherwise there could be issues with failed connections to other pods.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r489455381
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +135,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
-
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
affinityCtx = new ClientCacheAffinityContext(binary);
+ }
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
-
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
Review comment:
It will inside `channelsInit`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r487853045
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
Review comment:
To solve this problem we can store address as unresolved (InetSocketAddress#createUnresolved).
In your implementation initChannelHolders will be invoked in response reading thread on each topology change. I think address resolving in this thread can take some time and should be avoided.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504760666
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
+
+ } catch (Throwable ex) {
+ fut.completeExceptionally(ex);
+ return;
}
- if (err instanceof ClientConnectionException) {
- onChannelFailure(ch);
+ ClientChannel ch = chAndAttempts.getKey();
- if (failure == null)
- failure = (ClientConnectionException) err;
- else
- failure.addSuppressed(err);
+ ch
+ .serviceAsync(op, payloadWriter, payloadReader)
+ .handle((res, err) -> {
+ if (err == null) {
+ fut.complete(res);
Review comment:
Ah, I see in next comments. There is already NL, will not change it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r488760425
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -169,8 +167,10 @@
// No-op.
}
- for (ClientChannelHolder hld : channels)
- hld.closeChannel();
+ if (channels.get() != null) {
+ for (ClientChannelHolder hld: channels.get())
Review comment:
Sure, with current usages it may work, but the approach is fragile and hard to trace. See how you had to check every usage to make sure this logic is safe?
For volatiles and atomics it is a common practice to copy the value into a variable when you do multiple access and/or conditional logic:
* Easy to see that it is safe - less cognitive load
* Faster
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r489364226
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +135,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
-
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
affinityCtx = new ClientCacheAffinityContext(binary);
+ }
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
-
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
Review comment:
If partition awareness is enabled we still need to init default channel on startup
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -461,9 +418,8 @@ private void initAllChannelsAsync() {
* @param ch Channel.
*/
private void onTopologyChanged(ClientChannel ch) {
- if (partitionAwarenessEnabled && affinityCtx.updateLastTopologyVersion(ch.serverTopologyVersion(),
- ch.serverNodeId()))
- initAllChannelsAsync();
+ if (affinityCtx.updateLastTopologyVersion(ch.serverTopologyVersion(), ch.serverNodeId()))
+ channelsInit(true);
Review comment:
This method is invoked inside request processing thread, I don't like the idea of sync channel holders reinit (which can do additional sync http requests) on each topology change in this thread.
I think we should do it async if partition awareness is enabled. Additionally, we can do it sync in sending thread on channel failure and if we detect topology change before.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,16 +429,230 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean shouldStopChannelsReinit() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ synchronized void initChannelHolders(boolean force) {
+ List<ClientChannelHolder> holders = channels;
+
+ if (!force && holders != null)
+ return;
+
+ startChannelsReInit = System.currentTimeMillis();
+
+ // Enable parallel threads to schedule new init of channel holders.
+ scheduledChannelsReinit.set(false);
+
+ Set<InetSocketAddress> newAddrs = null;
+
+ if (clientCfg.getAddressesFinder() != null) {
+ String[] hostAddrs = clientCfg.getAddressesFinder().getServerAddresses();
+
+ if (hostAddrs.length == 0)
+ throw new ClientException("Empty addresses");
+
+ if (!Arrays.equals(hostAddrs, prevHostAddrs)) {
+ newAddrs = parsedAddresses(hostAddrs);
+ prevHostAddrs = hostAddrs;
+ }
+ } else if (holders == null)
+ newAddrs = parsedAddresses(clientCfg.getAddresses());
+
+ if (newAddrs == null) {
+ finishChannelsReInit = System.currentTimeMillis();
+ return;
+ }
+
+ Map<InetSocketAddress, ClientChannelHolder> curAddrs = Collections.emptyMap();
+ Set<InetSocketAddress> allAddrs = new HashSet<>(newAddrs);
+
+ if (holders != null) {
+ curAddrs = holders.stream()
+ .collect(Collectors.toMap(h -> h.chCfg.getAddress(), h -> h));
+
+ allAddrs.addAll(curAddrs.keySet());
+ }
+
+ List<ClientChannelHolder> reinitHolders = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currDfltHolder = null;
+ int idx = curChIdx;
+ if (idx != -1)
+ currDfltHolder = holders.get(idx);
+
+ for (InetSocketAddress addr : allAddrs) {
+ if (shouldStopChannelsReinit())
+ return;
+
+ // Obsolete addr, to be removed.
+ if (!newAddrs.contains(addr)) {
+ curAddrs.get(addr).close();
+
+ continue;
+ }
+
+ // Create new holders for new addrs.
+ if (!curAddrs.containsKey(addr)) {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ reinitHolders.add(hld);
+
+ continue;
+ }
+
+ // This holder is up to date.
+ ClientChannelHolder hld = curAddrs.get(addr);
+ reinitHolders.add(hld);
+ if (hld == currDfltHolder)
+ dfltChannelIdx = reinitHolders.size() - 1;
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(reinitHolders.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels = reinitHolders;
+ curChIdx = dfltChannelIdx;
+ }
+ finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+
+ finishChannelsReInit = System.currentTimeMillis();
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
+ if (!force && channels != null)
+ return;
+
+ // Skip if there is already channels reinit scheduled.
+ // Flag is set back when a thread comes in synchronized initChannelHolders.
+ if (scheduledChannelsReinit.compareAndSet(false, true)) {
+ initChannelHolders(force);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = hld != null ? hld.getOrCreateChannel() : null;
+
+ if (channel != null)
+ return function.apply(channel);
+
+ } catch (ClientConnectionException e) {
+ onChannelFailure(hld, channel);
+ }
+
+ return null;
+ }
+
+ /**
+ * Apply specified {@code function} on any of available channel.
+ */
+ private <T> T applyOnDefaultChannel(Function<ClientChannel, T> function) {
+ List<ClientChannelHolder> holders = channels;
+
+ if (holders == null)
+ throw new ClientException("Connections to nodes aren't initialized.");
+
+ int size = holders.size();
+
+ int attemptsLimit = clientCfg.getRetryLimit() > 0 ?
+ Math.min(clientCfg.getRetryLimit(), size) : size;
+
+ ClientConnectionException failure = null;
+
+ for (int attempt = 0; attempt < attemptsLimit; attempt++) {
+ ClientChannelHolder hld = null;
+ ClientChannel c = null;
+ try {
+ if (closed)
+ throw new ClientException("Channel is closed");
+
+ curChannelsGuard.readLock().lock();
+ try {
+ hld = channels.get(curChIdx);
+ } finally {
+ curChannelsGuard.readLock().unlock();
+ }
+
+ c = hld.getOrCreateChannel();
+ if (c != null)
+ return function.apply(c);
+ }
+ catch (ClientConnectionException e) {
+ if (failure == null)
+ failure = e;
+ else
+ failure.addSuppressed(e);
+
+ onChannelFailure(hld, c);
+ }
+ }
+
+ throw failure;
+ }
+
+ /**
+ * Try apply specified {@code function} on a channel corresponding to {@code tryNodeId}.
+ * If failed then apply the function on any available channel.
+ */
+ private <T> T apply(UUID tryNodeId, Function<ClientChannel, T> function) {
Review comment:
Let's rename it to something more meaningful (like another two `apply...` methods). For, example, `applyOnNodeChannelWithRetry` or something like that.
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
Review comment:
Let's use standard ports for thin client (10800). Yes, I know it's just a dummy address, but someone can understand it wrong during a brief test lookup.
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
+ */
+ @Test
+ public void testClientDiscoveryNodesJoin() throws Exception {
+ for (int i = 0; i < MAX_CLUSTER_SIZE; ++i) {
+ startGrid(i);
+ awaitPartitionMapExchange();
+
+ int[] workChannels = IntStream.rangeClosed(0, i).toArray();
+
+ if (i == 0)
+ initClient(getClientConfigurationWithDiscovery(), workChannels);
+ else {
+ detectTopologyChange();
+ awaitChannelsInit(workChannels);
+ }
+
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Test that client use channels to all running nodes while nodes stop
Review comment:
point at the end
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
+ */
+ @Test
+ public void testClientDiscoveryNodesJoin() throws Exception {
+ for (int i = 0; i < MAX_CLUSTER_SIZE; ++i) {
+ startGrid(i);
+ awaitPartitionMapExchange();
+
+ int[] workChannels = IntStream.rangeClosed(0, i).toArray();
+
+ if (i == 0)
+ initClient(getClientConfigurationWithDiscovery(), workChannels);
+ else {
+ detectTopologyChange();
+ awaitChannelsInit(workChannels);
+ }
+
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Test that client use channels to all running nodes while nodes stop
+ */
+ @Test
+ public void testClientDiscoveryNodesLeave() throws Exception {
Review comment:
We don't check discovery by this test at all. If we replace `getClientConfigurationWithDiscovery()` with `getClientConfiguration(0, 1, 2, 3)` result will be the same, test will pass. To check discovery I think we should exclude additional nodes by discovery (but not stop it) on topology change and check that this node is excluded from requests too.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -379,80 +351,65 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
return ranges.stream()
.flatMap(r -> IntStream
.rangeClosed(r.portFrom(), r.portTo()).boxed()
- .map(p -> new InetSocketAddress(r.host(), p))
+ .map(p -> InetSocketAddress.createUnresolved(r.host(), p))
)
- .collect(Collectors.toList());
- }
-
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
- try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
-
- throw e;
- }
- }
-
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ .collect(Collectors.toSet());
}
/**
- * On current channel failure.
+ * Roll current default channel if specified holder equals to it.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
- // There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
- // when current index was changed and no other wrong channel will be closed by current thread because
- // onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
+ try {
+ int idx = curChIdx;
+ List<ClientChannelHolder> holders = channels;
+
+ ClientChannelHolder dfltHld = holders.get(idx);
+ if (dfltHld == hld) {
+ idx += 1;
+ if (idx >= holders.size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (hld != null && ch != null && ch == hld.ch)
Review comment:
`hld` can't be null here
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +135,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
-
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
affinityCtx = new ClientCacheAffinityContext(binary);
+ }
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
-
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
Review comment:
Here and bellow, please use Ignite codestyle (see https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines `Semantic Units` section)
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -519,29 +689,44 @@ private boolean applyReconnectionThrottling() {
/**
* Get or create channel.
*/
- private synchronized ClientChannel getOrCreateChannel()
+ private ClientChannel getOrCreateChannel()
throws ClientConnectionException, ClientAuthenticationException, ClientProtocolError {
return getOrCreateChannel(false);
}
/**
* Get or create channel.
*/
- private synchronized ClientChannel getOrCreateChannel(boolean ignoreThrottling)
+ private ClientChannel getOrCreateChannel(boolean ignoreThrottling)
throws ClientConnectionException, ClientAuthenticationException, ClientProtocolError {
- if (ch == null) {
- if (!ignoreThrottling && applyReconnectionThrottling())
- throw new ClientConnectionException("Reconnect is not allowed due to applied throttling");
+ if (ch == null && !close) {
+ synchronized (this) {
+ if (close)
+ return null;
+
+ if (ch != null)
+ return ch;
- ch = chFactory.apply(chCfg);
+ if (!ignoreThrottling && applyReconnectionThrottling())
+ throw new ClientConnectionException("Reconnect is not allowed due to applied throttling");
- if (ch.serverNodeId() != null) {
- ch.addTopologyChangeListener(ReliableChannel.this::onTopologyChanged);
- ch.addNotificationListener(ReliableChannel.this);
+ ClientChannel channel = chFactory.apply(chCfg);
- nodeChannels.values().remove(this);
+ if (channel.serverNodeId() != null) {
+ channel.addTopologyChangeListener(ReliableChannel.this::onTopologyChanged);
+ channel.addNotificationListener(ReliableChannel.this);
- nodeChannels.putIfAbsent(ch.serverNodeId(), this);
+ if (serverNodeId == null)
+ serverNodeId = channel.serverNodeId();
+
+ if (serverNodeId != null && serverNodeId != channel.serverNodeId())
+ nodeChannels.remove(serverNodeId);
Review comment:
`nodeChannels.remove(serverNodeId, this)`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -561,5 +746,24 @@ private synchronized void closeChannel() {
ch = null;
}
}
+
+ /** Close holder. */
+ void close() {
+ close = true;
+ if (serverNodeId != null)
+ nodeChannels.remove(serverNodeId);
+
+ closeChannel();
+ }
+
+ /** Wheteher the holder is closed. For test purposes. */
+ boolean isClosed() {
+ return close;
+ }
+ }
+
+ /** Get holders reference. For test purposes. */
+ List<ClientChannelHolder> getChannelHolders() {
Review comment:
Let's make `channel` package private and use it in tests directly (or make `nodeChannels` private and create the same getter for it)
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -519,29 +689,44 @@ private boolean applyReconnectionThrottling() {
/**
* Get or create channel.
*/
- private synchronized ClientChannel getOrCreateChannel()
+ private ClientChannel getOrCreateChannel()
throws ClientConnectionException, ClientAuthenticationException, ClientProtocolError {
return getOrCreateChannel(false);
}
/**
* Get or create channel.
*/
- private synchronized ClientChannel getOrCreateChannel(boolean ignoreThrottling)
+ private ClientChannel getOrCreateChannel(boolean ignoreThrottling)
throws ClientConnectionException, ClientAuthenticationException, ClientProtocolError {
- if (ch == null) {
- if (!ignoreThrottling && applyReconnectionThrottling())
- throw new ClientConnectionException("Reconnect is not allowed due to applied throttling");
+ if (ch == null && !close) {
+ synchronized (this) {
+ if (close)
+ return null;
+
+ if (ch != null)
+ return ch;
- ch = chFactory.apply(chCfg);
+ if (!ignoreThrottling && applyReconnectionThrottling())
+ throw new ClientConnectionException("Reconnect is not allowed due to applied throttling");
- if (ch.serverNodeId() != null) {
- ch.addTopologyChangeListener(ReliableChannel.this::onTopologyChanged);
- ch.addNotificationListener(ReliableChannel.this);
+ ClientChannel channel = chFactory.apply(chCfg);
- nodeChannels.values().remove(this);
+ if (channel.serverNodeId() != null) {
+ channel.addTopologyChangeListener(ReliableChannel.this::onTopologyChanged);
+ channel.addNotificationListener(ReliableChannel.this);
- nodeChannels.putIfAbsent(ch.serverNodeId(), this);
+ if (serverNodeId == null)
Review comment:
This `if` is redundant, since you set serverNodeId later and the next `if` condition can't be true with this condition at the same time.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -561,5 +746,24 @@ private synchronized void closeChannel() {
ch = null;
}
}
+
+ /** Close holder. */
+ void close() {
+ close = true;
+ if (serverNodeId != null)
+ nodeChannels.remove(serverNodeId);
Review comment:
`nodeChannels.remove(serverNodeId, this)`
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** */
+ private void checkDoesNotReinit(ClientConfiguration ccfg) {
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ List<ReliableChannel.ClientChannelHolder> originalChannels = rc.getChannelHolders();
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+
+ assertSame(originalChannels, newChannels);
+ IntStream.range(0, 3).forEach(i -> {
+ assertSame(originalChannels.get(i), newChannels.get(i));
Review comment:
Didn't get this check. We already checked that references are the same. Isn't it?
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
+ */
+ @Test
+ public void testClientDiscoveryNodesJoin() throws Exception {
+ for (int i = 0; i < MAX_CLUSTER_SIZE; ++i) {
+ startGrid(i);
+ awaitPartitionMapExchange();
+
+ int[] workChannels = IntStream.rangeClosed(0, i).toArray();
+
+ if (i == 0)
+ initClient(getClientConfigurationWithDiscovery(), workChannels);
+ else {
+ detectTopologyChange();
+ awaitChannelsInit(workChannels);
+ }
+
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Test that client use channels to all running nodes while nodes stop
+ */
+ @Test
+ public void testClientDiscoveryNodesLeave() throws Exception {
+ startGrids(MAX_CLUSTER_SIZE);
+ awaitPartitionMapExchange();
+
+ initClient(getClientConfigurationWithDiscovery(), 0, 1, 2, 3);
+ detectTopologyChange();
+
+ for (int i = MAX_CLUSTER_SIZE - 1; i != 0; i--) {
+ int[] workChannels = IntStream.range(0, i).toArray();
+
+ channels[i] = null;
+ stopGrid(i);
+ awaitPartitionMapExchange();
+ detectTopologyChange();
+
+ awaitChannelsInit(workChannels);
Review comment:
All channels already inited after initClient, what else do we wait here?
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
+ */
+ @Test
+ public void testClientDiscoveryNodesJoin() throws Exception {
+ for (int i = 0; i < MAX_CLUSTER_SIZE; ++i) {
+ startGrid(i);
+ awaitPartitionMapExchange();
+
+ int[] workChannels = IntStream.rangeClosed(0, i).toArray();
+
+ if (i == 0)
+ initClient(getClientConfigurationWithDiscovery(), workChannels);
+ else {
+ detectTopologyChange();
+ awaitChannelsInit(workChannels);
+ }
+
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Test that client use channels to all running nodes while nodes stop
+ */
+ @Test
+ public void testClientDiscoveryNodesLeave() throws Exception {
+ startGrids(MAX_CLUSTER_SIZE);
+ awaitPartitionMapExchange();
+
+ initClient(getClientConfigurationWithDiscovery(), 0, 1, 2, 3);
+ detectTopologyChange();
+
+ for (int i = MAX_CLUSTER_SIZE - 1; i != 0; i--) {
+ int[] workChannels = IntStream.range(0, i).toArray();
+
+ channels[i] = null;
+ stopGrid(i);
+ awaitPartitionMapExchange();
+ detectTopologyChange();
+
+ awaitChannelsInit(workChannels);
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Checks that each request goes to right node.
+ */
+ private void testPartitionAwareness(int... chIdxs) {
+ ClientCache<Object, Object> clientCache = client.cache(PART_CACHE_NAME);
+ IgniteInternalCache<Object, Object> igniteCache = grid(0).context().cache().cache(PART_CACHE_NAME);
+
+ Map<TestTcpClientChannel, Boolean> channelHits = Arrays.stream(chIdxs).boxed()
+ .collect(Collectors.toMap(idx -> channels[idx], idx -> false));
+
+ for (int i = 0; i < KEY_CNT; i++) {
+ TestTcpClientChannel opCh = affinityChannel(i, igniteCache);
+
+ clientCache.put(i, i);
+
+ if (i == 0)
+ assertOpOnChannel(dfltCh, ClientOperation.CACHE_PARTITIONS);
+
+ assertOpOnChannel(opCh, ClientOperation.CACHE_PUT);
+ assertTrue(channelHits.containsKey(opCh));
+
+ channelHits.compute(opCh, (c, old) -> true);
+ }
+
+ assertFalse(channelHits.containsValue(false));
+ }
+
+ /**
+ * Provide ClientConfiguration with addrResolver that find all alive nodes
+ */
+ private ClientConfiguration getClientConfigurationWithDiscovery() {
+ ClientAddressFinder addrFinder = () ->
+ IgnitionEx.allGrids().stream().map(node -> {
+ int port = (Integer) node.cluster().localNode().attributes().get(CLIENT_LISTENER_PORT);
+ return "127.0.0.1:" + port;
+ }).toArray(String[]::new);
+
+ return new ClientConfiguration()
+ .setAddressesFinder(addrFinder)
+ .setPartitionAwarenessEnabled(true);
+ }
+
+ /**
+ * Trigger client to detect topology change
+ */
+ private void detectTopologyChange() {
+ // Send non-affinity request to detect topology change.
+ initDefaultChannel();
+ client.getOrCreateCache(PART_CACHE_NAME);
Review comment:
Why do we need a second request? Topology change already should be detected after `initDefaultChannel` call
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** */
+ private void checkDoesNotReinit(ClientConfiguration ccfg) {
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ List<ReliableChannel.ClientChannelHolder> originalChannels = rc.getChannelHolders();
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+
+ assertSame(originalChannels, newChannels);
+ IntStream.range(0, 3).forEach(i -> {
+ assertSame(originalChannels.get(i), newChannels.get(i));
+ assertFalse(originalChannels.get(i).isClosed());
+ });
+ assertEquals(3, newChannels.size());
+ }
+
+ /** Checks that node channels are persisted if channels are reinit with static address configuration. */
+ @Test
+ public void testNodeChannelsAreNotCleaned() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ // Trigger TestClientChannel creation.
+ rc.service(null, null, null);
+
+ assertEquals(1, rc.nodeChannels.size());
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+
+ assertEquals(1, rc.nodeChannels.size());
+ }
+
+ /** Checks that channels are changed (add new, remove old) and close channels if reinitialization performed. */
+ @Test
+ public void testDynamicAddressReinitializedCorrectly() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8003"));
+
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+
+ List<ReliableChannel.ClientChannelHolder> originChannels = Collections.unmodifiableList(rc.getChannelHolders());
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+
+ List<ReliableChannel.ClientChannelHolder> closedChannels = originChannels.stream()
+ .filter(r -> r.isClosed())
+ .collect(Collectors.toList());
+
+ assertEquals(2, closedChannels.size());
+
+ List<ReliableChannel.ClientChannelHolder> reuseChannel = originChannels.stream()
+ .filter(c -> !c.isClosed())
+ .collect(Collectors.toList());
+
+ assertEquals(1, reuseChannel.size());
+
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+ assertEquals(2, newChannels.size());
+ assertTrue(newChannels.get(0) == reuseChannel.get(0) || newChannels.get(1) == reuseChannel.get(0));
Review comment:
newChannels.contains?
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
Review comment:
point at the end
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** */
+ private void checkDoesNotReinit(ClientConfiguration ccfg) {
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ List<ReliableChannel.ClientChannelHolder> originalChannels = rc.getChannelHolders();
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+
+ assertSame(originalChannels, newChannels);
+ IntStream.range(0, 3).forEach(i -> {
Review comment:
IMO it's more readable to use `for loop` here
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** */
+ private void checkDoesNotReinit(ClientConfiguration ccfg) {
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ List<ReliableChannel.ClientChannelHolder> originalChannels = rc.getChannelHolders();
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+
+ assertSame(originalChannels, newChannels);
+ IntStream.range(0, 3).forEach(i -> {
+ assertSame(originalChannels.get(i), newChannels.get(i));
+ assertFalse(originalChannels.get(i).isClosed());
+ });
+ assertEquals(3, newChannels.size());
+ }
+
+ /** Checks that node channels are persisted if channels are reinit with static address configuration. */
+ @Test
+ public void testNodeChannelsAreNotCleaned() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ // Trigger TestClientChannel creation.
+ rc.service(null, null, null);
+
+ assertEquals(1, rc.nodeChannels.size());
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+
+ assertEquals(1, rc.nodeChannels.size());
+ }
+
+ /** Checks that channels are changed (add new, remove old) and close channels if reinitialization performed. */
+ @Test
+ public void testDynamicAddressReinitializedCorrectly() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8003"));
+
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+
+ List<ReliableChannel.ClientChannelHolder> originChannels = Collections.unmodifiableList(rc.getChannelHolders());
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+
+ List<ReliableChannel.ClientChannelHolder> closedChannels = originChannels.stream()
+ .filter(r -> r.isClosed())
+ .collect(Collectors.toList());
+
+ assertEquals(2, closedChannels.size());
Review comment:
`assertEquals(2, F.size(originChannels, r -> r.isClosed()));`
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
Review comment:
It's counterintuitive without looking at `TestAddressFinder` implementation that we use the same set of addresses. Let's make TestAddressFinder more generic, use some Queue inside, and use it something like this:
`new TestAddressFinder().add("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002").add("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002")`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r483469994
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/kubernetes/KubernetesServiceAddressResolver.java
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.kubernetes;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.URL;
+import java.net.UnknownHostException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.security.SecureRandom;
+import java.security.cert.X509Certificate;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import javax.net.ssl.HostnameVerifier;
+import javax.net.ssl.HttpsURLConnection;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSession;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * The class is responsible to fetch list of IP address for all pods that runs the specified kubernetes service.
+ */
+public class KubernetesServiceAddressResolver {
Review comment:
The fact that it is being used by other classes does not require us to put it in a "public" package. Can we move this to `org.apache.ignite.internal.kubernetes` package?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r487720713
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/client/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.client;
+
+import java.net.InetAddress;
+import java.util.function.Supplier;
+import org.apache.ignite.internal.kubernetes.connection.KubernetesServiceAddressResolver;
+import org.apache.ignite.kubernetes.configuration.KubernetesConnectionConfiguration;
+
+/**
+ * Address finder for automatic lookup of Ignite server nodes running in Kubernetes environment. All Ignite nodes have
+ * to be deployed as Kubernetes pods in order to be found. Applications and Ignite nodes running outside of Kubernetes
+ * will not be able to reach the containerized counterparts.
+ * <p>
+ * The implementation is based on a distinct Kubernetes service. The name of the service must be set with
+ * {@code KubernetesConnectionConfiguration}. As for Ignite pods, it's recommended to label them in such a way that
+ * the service will target only server nodes.
+ * <p>
+ * The address finder, in its turn, will call this service to retrieve Ignite pods IP addresses. The port will be
+ * set later within {@link IgniteClient}. Make sure that all Ignite pods occupy a similar ClientConnector port,
+ * otherwise they will not be able to connect each other using this address finder.
+ * <p>
+ */
+public class ThinClientKubernetesAddressFinder implements Supplier<String[]> {
+ /** Kubernetes service address resolver. */
+ private final KubernetesServiceAddressResolver resolver;
+
+ /** Constructor */
+ public ThinClientKubernetesAddressFinder(KubernetesConnectionConfiguration cfg) {
+ resolver = new KubernetesServiceAddressResolver(cfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override public String[] get() {
+ return resolver
+ .getServiceAddresses()
+ .stream().map(InetAddress::getHostAddress)
+ .toArray(String[]::new);
Review comment:
There will be 100 holders created for each host (and there will be 100 retries to connect for each host) if port is not set explicitly (see ReliableChannel#parseAddresses). Can we also add ports to this array?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504759762
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
+
+ } catch (Throwable ex) {
+ fut.completeExceptionally(ex);
+ return;
}
- if (err instanceof ClientConnectionException) {
- onChannelFailure(ch);
+ ClientChannel ch = chAndAttempts.getKey();
- if (failure == null)
- failure = (ClientConnectionException) err;
- else
- failure.addSuppressed(err);
+ ch
+ .serviceAsync(op, payloadWriter, payloadReader)
+ .handle((res, err) -> {
+ if (err == null) {
+ fut.complete(res);
Review comment:
Maybe you mean "Redundant NL"? Will fIx it this way
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r501246249
##########
File path: modules/core/src/main/java/org/apache/ignite/client/ClientAddressFinder.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.client;
+
+/**
+ * This interface provides a list of addresses of Ignite server nodes within a cluster. Thin client uses the list to
+ * route user requests. There are cases when the list is not static, for example in cloud environment. In such cases
+ * addresses of nodes and/or number of server nodes can change. Implementation of this interface should handle these.
+ *
+ * Ignite waites for a topology change to trigger {@link #getAddresses()}. There are two modes of how soon Ignite calls
+ * it, lazy and eager, depending on whether the partition awareness feature is enabled. If the feature is enabled then
+ * Ignite calls the method for every topology change. Otherwise Ignite uses previous addresses until a first failure.
+ *
+ * {@link org.apache.ignite.configuration.ClientConfiguration#setPartitionAwarenessEnabled(boolean)}
+ * {@link org.apache.ignite.configuration.ClientConfiguration#setAddressesFinder(ClientAddressFinder)}
+ */
+public interface ClientAddressFinder {
+ /**
+ * Get addresses of Ignite server nodes within a cluster. An address can be IPv4 address or hostname, with or
Review comment:
Agree, I've created a ticket https://issues.apache.org/jira/browse/IGNITE-13555
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r497490174
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -561,5 +746,24 @@ private synchronized void closeChannel() {
ch = null;
}
}
+
+ /** Close holder. */
+ void close() {
+ close = true;
+ if (serverNodeId != null)
+ nodeChannels.remove(serverNodeId);
+
+ closeChannel();
+ }
+
+ /** Wheteher the holder is closed. For test purposes. */
+ boolean isClosed() {
+ return close;
+ }
+ }
+
+ /** Get holders reference. For test purposes. */
+ List<ClientChannelHolder> getChannelHolders() {
Review comment:
fixed.
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504966233
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
Review comment:
I tried rewrite this code, but it only becomes more complicated (lambdas starts to complain about non-final scope, etc.). S, I suggest another approach with introducing a callback to store attempts counter in an external value (push commit). WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r488665920
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -493,6 +520,25 @@ public ClientConfiguration setReconnectThrottlingRetries(int reconnectThrottling
return this;
}
+ /**
+ * Get retry limit.
+ */
+ public int getRetryLimit() {
+ return retryLimit;
+ }
+
+ /**
+ * Try use limited number of channels to send a request if default channel is not responding.
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482818928
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +116,33 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /**
+ * Try use other limited number of channels to send a request if default channel is not responding.
+ * 0 means try use all configured channels before fail.
+ */
+ private int channelsAttemptsLimit = 0;
Review comment:
I introduced this variable because number of configured nodes dynamically changes. Earlier user could just configure addresses with limited number of nodes. With k8s configuration number of nodes changes. And in case of large number of non-responding nodes (cluster is down, for example) without this limit client will wait for a long time for an exception.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r484050018
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/kubernetes/KubernetesConnectorConfigurator.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.kubernetes;
+
+/**
+ * Interface provide methods to configure Kubernetes connection.
+ */
+public interface KubernetesConnectorConfigurator {
Review comment:
I've implemented the second option. I think, it got prettier.
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/kubernetes/KubernetesServiceAddressResolver.java
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.kubernetes;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.URL;
+import java.net.UnknownHostException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.security.SecureRandom;
+import java.security.cert.X509Certificate;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import javax.net.ssl.HostnameVerifier;
+import javax.net.ssl.HttpsURLConnection;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSession;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * The class is responsible to fetch list of IP address for all pods that runs the specified kubernetes service.
+ */
+public class KubernetesServiceAddressResolver {
Review comment:
Moved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r494265337
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** */
+ private void checkDoesNotReinit(ClientConfiguration ccfg) {
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ List<ReliableChannel.ClientChannelHolder> originalChannels = rc.getChannelHolders();
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+
+ assertSame(originalChannels, newChannels);
+ IntStream.range(0, 3).forEach(i -> {
+ assertSame(originalChannels.get(i), newChannels.get(i));
Review comment:
FIxed, add test for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r488675595
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/client/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.client;
+
+import java.net.InetAddress;
+import java.util.function.Supplier;
+import org.apache.ignite.internal.kubernetes.connection.KubernetesServiceAddressResolver;
+import org.apache.ignite.kubernetes.configuration.KubernetesConnectionConfiguration;
+
+/**
+ * Address finder for automatic lookup of Ignite server nodes running in Kubernetes environment. All Ignite nodes have
+ * to be deployed as Kubernetes pods in order to be found. Applications and Ignite nodes running outside of Kubernetes
+ * will not be able to reach the containerized counterparts.
+ * <p>
+ * The implementation is based on a distinct Kubernetes service. The name of the service must be set with
+ * {@code KubernetesConnectionConfiguration}. As for Ignite pods, it's recommended to label them in such a way that
+ * the service will target only server nodes.
+ * <p>
+ * The address finder, in its turn, will call this service to retrieve Ignite pods IP addresses. The port will be
+ * set later within {@link IgniteClient}. Make sure that all Ignite pods occupy a similar ClientConnector port,
+ * otherwise they will not be able to connect each other using this address finder.
+ * <p>
+ */
+public class ThinClientKubernetesAddressFinder implements Supplier<String[]> {
+ /** Kubernetes service address resolver. */
+ private final KubernetesServiceAddressResolver resolver;
+
+ /** Constructor */
+ public ThinClientKubernetesAddressFinder(KubernetesConnectionConfiguration cfg) {
+ resolver = new KubernetesServiceAddressResolver(cfg);
+ }
+
+ /** {@inheritDoc} */
+ @Override public String[] get() {
+ return resolver
+ .getServiceAddresses()
+ .stream().map(InetAddress::getHostAddress)
+ .toArray(String[]::new);
Review comment:
Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482855961
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannelFacade.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.ignite.IgniteBinary;
+import org.apache.ignite.client.ClientAuthenticationException;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+
+/**
+ * Communication channel with failover and partition awareness.
+ */
+final class ReliableChannelFacade implements AutoCloseable {
Review comment:
I think that `ReliableChannel` was overloaded, it was responsible for 2 things the same time: communication and affinity awareness. Due to that there were some non-intuitive conditions for me while I debug the code. So I split the original class on 2 parts:
1. `ReliableChannel` now is responsible for sending **any** request to Ignite nodes. If some channel is failure it tries to use another one, roll channels, etc. It just provides infrastructure for communication between client and servers.
2. `ReliableChannelFacade` provides logic with affinity awareness: send **affinity** or user requests, choose affinity node for a request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r501125491
##########
File path: modules/core/src/main/java/org/apache/ignite/client/ClientAddressFinder.java
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.client;
+
+/**
+ * This interface provides a list of addresses of Ignite server nodes within a cluster. Thin client uses the list to
+ * route user requests. There are cases when the list is not static, for example in cloud environment. In such cases
+ * addresses of nodes and/or number of server nodes can change. Implementation of this interface should handle these.
+ *
+ * {@link org.apache.ignite.configuration.ClientConfiguration#setAddressesFinder(ClientAddressFinder)}
+ */
+public interface ClientAddressFinder {
+ /** @return Addresses of Ignite server nodes within a cluster. */
+ public String[] getServerAddresses();
Review comment:
Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r493533702
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** */
+ private void checkDoesNotReinit(ClientConfiguration ccfg) {
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ List<ReliableChannel.ClientChannelHolder> originalChannels = rc.getChannelHolders();
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+
+ assertSame(originalChannels, newChannels);
+ IntStream.range(0, 3).forEach(i -> {
+ assertSame(originalChannels.get(i), newChannels.get(i));
+ assertFalse(originalChannels.get(i).isClosed());
+ });
+ assertEquals(3, newChannels.size());
+ }
+
+ /** Checks that node channels are persisted if channels are reinit with static address configuration. */
+ @Test
+ public void testNodeChannelsAreNotCleaned() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ // Trigger TestClientChannel creation.
+ rc.service(null, null, null);
+
+ assertEquals(1, rc.nodeChannels.size());
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+
+ assertEquals(1, rc.nodeChannels.size());
+ }
+
+ /** Checks that channels are changed (add new, remove old) and close channels if reinitialization performed. */
+ @Test
+ public void testDynamicAddressReinitializedCorrectly() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8003"));
+
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+
+ List<ReliableChannel.ClientChannelHolder> originChannels = Collections.unmodifiableList(rc.getChannelHolders());
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+
+ List<ReliableChannel.ClientChannelHolder> closedChannels = originChannels.stream()
+ .filter(r -> r.isClosed())
+ .collect(Collectors.toList());
+
+ assertEquals(2, closedChannels.size());
+
+ List<ReliableChannel.ClientChannelHolder> reuseChannel = originChannels.stream()
+ .filter(c -> !c.isClosed())
+ .collect(Collectors.toList());
+
+ assertEquals(1, reuseChannel.size());
+
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+ assertEquals(2, newChannels.size());
+ assertTrue(newChannels.get(0) == reuseChannel.get(0) || newChannels.get(1) == reuseChannel.get(0));
Review comment:
contains invoke `equals` but I check references.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r494969192
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -461,9 +418,8 @@ private void initAllChannelsAsync() {
* @param ch Channel.
*/
private void onTopologyChanged(ClientChannel ch) {
- if (partitionAwarenessEnabled && affinityCtx.updateLastTopologyVersion(ch.serverTopologyVersion(),
- ch.serverNodeId()))
- initAllChannelsAsync();
+ if (affinityCtx.updateLastTopologyVersion(ch.serverTopologyVersion(), ch.serverNodeId()))
+ channelsInit(true);
Review comment:
Fair. Fixed it. Change signature of `initChannelHolders`, now it's not void but returns boolean (flag of interruption). Otherwise for disabled awareness and continuous topology events there would be a chances for StackOverflow, as channel failure trigger recursively `applyOnDefaultChannel` to enable default channel.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r497487790
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** */
+ private void checkDoesNotReinit(ClientConfiguration ccfg) {
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ List<ReliableChannel.ClientChannelHolder> originalChannels = rc.getChannelHolders();
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+
+ assertSame(originalChannels, newChannels);
+ IntStream.range(0, 3).forEach(i -> {
+ assertSame(originalChannels.get(i), newChannels.get(i));
Review comment:
Fair, miss it. Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r486089622
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
Review comment:
replaced with Set
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +116,30 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return addrFinder.get();
Review comment:
fixed
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/client/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.client;
+
+import java.net.InetAddress;
+import java.util.function.Supplier;
+import org.apache.ignite.internal.kubernetes.connection.KubernetesServiceAddressResolver;
+import org.apache.ignite.kubernetes.configuration.KubernetesConnectionConfiguration;
+
+/**
+ * Address finder for automatic lookup of Ignite server nodes running in Kubernetes environment. All Ignite nodes have
+ * to be deployed as Kubernetes pods in order to be found. Applications and Ignite nodes running outside of Kubernetes
+ * will not be able to reach the containerized counterparts.
+ * <p>
+ * The implementation is based on a distinct Kubernetes service. The name of the service must be set with
+ * {@code KubernetesConnectionConfiguration}. As for Ignite pods, it's recommended to label them in such a way that
+ * the service will target only server nodes.
+ * <p>
+ * The address finder, in its turn, will call this service to retrieve Ignite pods IP addresses. The port will be
+ * set later within {@link IgniteClient}. Make sure that all Ignite pods occupy a similar ClientConnector port,
+ * otherwise they will not be able to connect each other using this address finder.
+ * <p>
+ */
+public class ThinClientKubernetesAddressFinder implements Supplier<String[]> {
+ /** Kubernetes connection configuration */
+ private final KubernetesConnectionConfiguration cfg;
+
+ /** Constructor */
+ public ThinClientKubernetesAddressFinder(KubernetesConnectionConfiguration cfg) {
+ this.cfg = cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override public String[] get() {
+ return new KubernetesServiceAddressResolver(cfg)
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482814669
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(
+ new ClientChannelConfiguration(clientCfg, addr), chFactory, onChannelCreate, onChannelClose);
+
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
- affinityCtx = new ClientCacheAffinityContext(binary);
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
- ClientConnectionException lastEx = null;
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
- for (int i = 0; i < channels.length; i++) {
+ curChannelsGuard.writeLock().lock();
Review comment:
This lock is around two variables that must be in sync: channels (AtomicReference) and curChIdx.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r488675490
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
Review comment:
Fixed. Looks simplier.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r483151133
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +116,33 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /**
+ * Try use other limited number of channels to send a request if default channel is not responding.
+ * 0 means try use all configured channels before fail.
+ */
+ private int channelsAttemptsLimit = 0;
Review comment:
Rename to retryLimit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r487017283
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +126,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
-
- affinityCtx = new ClientCacheAffinityContext(binary);
-
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
+ affCtx = new ClientCacheAffinityContext(binary);
+ }
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
+ applyOnDefaultChannel(channel -> {
+ // do nothing, just trigger channel connection.
Review comment:
Here and below, please use Ignite code style for comments (start with uppercase later, end with a point).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
Review comment:
Can we skip further steps if `clientCfg.getAddresses()` is not changed? (for example, in case of static addresses configuration you will always reinit holders, but it's redundant)
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -356,7 +318,7 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
/**
* @return host:port_range address lines parsed as {@link InetSocketAddress}.
*/
- private static List<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
+ private Set<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
Review comment:
Let's keep it `static`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
Review comment:
I'm not sure that replacing `List` to `Set` it's a good idea here (perhaps you should just copy list to set here locally). Using `List` you can solve some user problems which can't be solved with `Set`. For example, using `list` you can make some server nodes to be used more often than another. Also, by having a single duplicated address you can workaround some known issues.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
Review comment:
`new ArrayList<>()` -> `Collections.emptyList()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(
+ new ClientChannelConfiguration(clientCfg, addr), chFactory, onChannelCreate, onChannelClose);
+
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
- affinityCtx = new ClientCacheAffinityContext(binary);
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
- ClientConnectionException lastEx = null;
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
- for (int i = 0; i < channels.length; i++) {
+ curChannelsGuard.writeLock().lock();
Review comment:
Agree with Pavel, we don't need `AtomicReference`, it's too verbose and we don't use any atomic features (no CAS operations, only get and set). Can be replaced with volatile variable (List or array).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
Review comment:
`force` flag is only set when partition awareness is used, but I think sometimes we should reinit holders even without partition awareness. For example, if we use Kubernetes discovery and new servers were started, the client will use only addresses of server nodes that were available at the time of client start.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
Review comment:
Perhaps code will be much more simple with regular loop over holders without streams and intermediate structures.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +126,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
-
- affinityCtx = new ClientCacheAffinityContext(binary);
-
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
+ affCtx = new ClientCacheAffinityContext(binary);
+ }
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
+ applyOnDefaultChannel(channel -> {
+ // do nothing, just trigger channel connection.
Review comment:
Here and below, please use Ignite code style for comments (start with uppercase later, end with a point).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
Review comment:
Can we skip further steps if `clientCfg.getAddresses()` is not changed? (for example, in case of static addresses configuration you will always reinit holders, but it's redundant)
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -356,7 +318,7 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
/**
* @return host:port_range address lines parsed as {@link InetSocketAddress}.
*/
- private static List<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
+ private Set<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
Review comment:
Let's keep it `static`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
Review comment:
I'm not sure that replacing `List` to `Set` it's a good idea here (perhaps you should just copy list to set here locally). Using `List` you can solve some user problems which can't be solved with `Set`. For example, using `list` you can make some server nodes to be used more often than another. Also, by having a single duplicated address you can workaround some known issues.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
Review comment:
`new ArrayList<>()` -> `Collections.emptyList()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(
+ new ClientChannelConfiguration(clientCfg, addr), chFactory, onChannelCreate, onChannelClose);
+
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
- affinityCtx = new ClientCacheAffinityContext(binary);
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
- ClientConnectionException lastEx = null;
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
- for (int i = 0; i < channels.length; i++) {
+ curChannelsGuard.writeLock().lock();
Review comment:
Agree with Pavel, we don't need `AtomicReference`, it's too verbose and we don't use any atomic features (no CAS operations, only get and set). Can be replaced with volatile variable (List or array).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
Review comment:
`force` flag is only set when partition awareness is used, but I think sometimes we should reinit holders even without partition awareness. For example, if we use Kubernetes discovery and new servers were started, the client will use only addresses of server nodes that were available at the time of client start.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
Review comment:
Perhaps code will be much more simple with regular loop over holders without streams and intermediate structures.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +126,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
-
- affinityCtx = new ClientCacheAffinityContext(binary);
-
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
+ affCtx = new ClientCacheAffinityContext(binary);
+ }
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
+ applyOnDefaultChannel(channel -> {
+ // do nothing, just trigger channel connection.
Review comment:
Here and below, please use Ignite code style for comments (start with uppercase later, end with a point).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
Review comment:
Can we skip further steps if `clientCfg.getAddresses()` is not changed? (for example, in case of static addresses configuration you will always reinit holders, but it's redundant)
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -356,7 +318,7 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
/**
* @return host:port_range address lines parsed as {@link InetSocketAddress}.
*/
- private static List<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
+ private Set<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
Review comment:
Let's keep it `static`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
Review comment:
I'm not sure that replacing `List` to `Set` it's a good idea here (perhaps you should just copy list to set here locally). Using `List` you can solve some user problems which can't be solved with `Set`. For example, using `list` you can make some server nodes to be used more often than another. Also, by having a single duplicated address you can workaround some known issues.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
Review comment:
`new ArrayList<>()` -> `Collections.emptyList()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(
+ new ClientChannelConfiguration(clientCfg, addr), chFactory, onChannelCreate, onChannelClose);
+
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
- affinityCtx = new ClientCacheAffinityContext(binary);
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
- ClientConnectionException lastEx = null;
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
- for (int i = 0; i < channels.length; i++) {
+ curChannelsGuard.writeLock().lock();
Review comment:
Agree with Pavel, we don't need `AtomicReference`, it's too verbose and we don't use any atomic features (no CAS operations, only get and set). Can be replaced with volatile variable (List or array).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
Review comment:
`force` flag is only set when partition awareness is used, but I think sometimes we should reinit holders even without partition awareness. For example, if we use Kubernetes discovery and new servers were started, the client will use only addresses of server nodes that were available at the time of client start.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
Review comment:
Perhaps code will be much more simple with regular loop over holders without streams and intermediate structures.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +126,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
-
- affinityCtx = new ClientCacheAffinityContext(binary);
-
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
+ affCtx = new ClientCacheAffinityContext(binary);
+ }
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
+ applyOnDefaultChannel(channel -> {
+ // do nothing, just trigger channel connection.
Review comment:
Here and below, please use Ignite code style for comments (start with uppercase later, end with a point).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
Review comment:
Can we skip further steps if `clientCfg.getAddresses()` is not changed? (for example, in case of static addresses configuration you will always reinit holders, but it's redundant)
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -356,7 +318,7 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
/**
* @return host:port_range address lines parsed as {@link InetSocketAddress}.
*/
- private static List<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
+ private Set<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
Review comment:
Let's keep it `static`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
Review comment:
I'm not sure that replacing `List` to `Set` it's a good idea here (perhaps you should just copy list to set here locally). Using `List` you can solve some user problems which can't be solved with `Set`. For example, using `list` you can make some server nodes to be used more often than another. Also, by having a single duplicated address you can workaround some known issues.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
Review comment:
`new ArrayList<>()` -> `Collections.emptyList()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(
+ new ClientChannelConfiguration(clientCfg, addr), chFactory, onChannelCreate, onChannelClose);
+
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
- affinityCtx = new ClientCacheAffinityContext(binary);
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
- ClientConnectionException lastEx = null;
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
- for (int i = 0; i < channels.length; i++) {
+ curChannelsGuard.writeLock().lock();
Review comment:
Agree with Pavel, we don't need `AtomicReference`, it's too verbose and we don't use any atomic features (no CAS operations, only get and set). Can be replaced with volatile variable (List or array).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
Review comment:
`force` flag is only set when partition awareness is used, but I think sometimes we should reinit holders even without partition awareness. For example, if we use Kubernetes discovery and new servers were started, the client will use only addresses of server nodes that were available at the time of client start.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
Review comment:
Perhaps code will be much more simple with regular loop over holders without streams and intermediate structures.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +126,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
-
- affinityCtx = new ClientCacheAffinityContext(binary);
-
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
+ affCtx = new ClientCacheAffinityContext(binary);
+ }
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
+ applyOnDefaultChannel(channel -> {
+ // do nothing, just trigger channel connection.
Review comment:
Here and below, please use Ignite code style for comments (start with uppercase later, end with a point).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
Review comment:
Can we skip further steps if `clientCfg.getAddresses()` is not changed? (for example, in case of static addresses configuration you will always reinit holders, but it's redundant)
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -356,7 +318,7 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
/**
* @return host:port_range address lines parsed as {@link InetSocketAddress}.
*/
- private static List<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
+ private Set<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
Review comment:
Let's keep it `static`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
Review comment:
I'm not sure that replacing `List` to `Set` it's a good idea here (perhaps you should just copy list to set here locally). Using `List` you can solve some user problems which can't be solved with `Set`. For example, using `list` you can make some server nodes to be used more often than another. Also, by having a single duplicated address you can workaround some known issues.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
Review comment:
`new ArrayList<>()` -> `Collections.emptyList()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(
+ new ClientChannelConfiguration(clientCfg, addr), chFactory, onChannelCreate, onChannelClose);
+
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
- affinityCtx = new ClientCacheAffinityContext(binary);
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
- ClientConnectionException lastEx = null;
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
- for (int i = 0; i < channels.length; i++) {
+ curChannelsGuard.writeLock().lock();
Review comment:
Agree with Pavel, we don't need `AtomicReference`, it's too verbose and we don't use any atomic features (no CAS operations, only get and set). Can be replaced with volatile variable (List or array).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
Review comment:
`force` flag is only set when partition awareness is used, but I think sometimes we should reinit holders even without partition awareness. For example, if we use Kubernetes discovery and new servers were started, the client will use only addresses of server nodes that were available at the time of client start.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
Review comment:
Perhaps code will be much more simple with regular loop over holders without streams and intermediate structures.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r494268992
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -461,9 +418,8 @@ private void initAllChannelsAsync() {
* @param ch Channel.
*/
private void onTopologyChanged(ClientChannel ch) {
- if (partitionAwarenessEnabled && affinityCtx.updateLastTopologyVersion(ch.serverTopologyVersion(),
- ch.serverNodeId()))
- initAllChannelsAsync();
+ if (affinityCtx.updateLastTopologyVersion(ch.serverTopologyVersion(), ch.serverNodeId()))
+ channelsInit(true);
Review comment:
If partition awareness is disabled and a new node is joined it's fine to work with the current node until failure (we don't need to find new nodes unless we disconnected from the current).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r497407082
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** */
+ private void checkDoesNotReinit(ClientConfiguration ccfg) {
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ List<ReliableChannel.ClientChannelHolder> originalChannels = rc.getChannelHolders();
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+
+ assertSame(originalChannels, newChannels);
+ IntStream.range(0, 3).forEach(i -> {
+ assertSame(originalChannels.get(i), newChannels.get(i));
+ assertFalse(originalChannels.get(i).isClosed());
+ });
+ assertEquals(3, newChannels.size());
+ }
+
+ /** Checks that node channels are persisted if channels are reinit with static address configuration. */
+ @Test
+ public void testNodeChannelsAreNotCleaned() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ // Trigger TestClientChannel creation.
+ rc.service(null, null, null);
+
+ assertEquals(1, rc.nodeChannels.size());
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+
+ assertEquals(1, rc.nodeChannels.size());
+ }
+
+ /** Checks that channels are changed (add new, remove old) and close channels if reinitialization performed. */
+ @Test
+ public void testDynamicAddressReinitializedCorrectly() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8003"));
+
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+
+ List<ReliableChannel.ClientChannelHolder> originChannels = Collections.unmodifiableList(rc.getChannelHolders());
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+
+ List<ReliableChannel.ClientChannelHolder> closedChannels = originChannels.stream()
+ .filter(r -> r.isClosed())
+ .collect(Collectors.toList());
+
+ assertEquals(2, closedChannels.size());
+
+ List<ReliableChannel.ClientChannelHolder> reuseChannel = originChannels.stream()
+ .filter(c -> !c.isClosed())
+ .collect(Collectors.toList());
+
+ assertEquals(1, reuseChannel.size());
+
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+ assertEquals(2, newChannels.size());
+ assertTrue(newChannels.get(0) == reuseChannel.get(0) || newChannels.get(1) == reuseChannel.get(0));
Review comment:
Currently it is, but the behavior may change in future and test will start check different case (we'll be lucky enough it will fail). Also I believe that explicit check is better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r497485647
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
Review comment:
Fixed, add tests for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r494250150
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
+ */
+ @Test
+ public void testClientDiscoveryNodesJoin() throws Exception {
+ for (int i = 0; i < MAX_CLUSTER_SIZE; ++i) {
+ startGrid(i);
+ awaitPartitionMapExchange();
+
+ int[] workChannels = IntStream.rangeClosed(0, i).toArray();
+
+ if (i == 0)
+ initClient(getClientConfigurationWithDiscovery(), workChannels);
+ else {
+ detectTopologyChange();
+ awaitChannelsInit(workChannels);
+ }
+
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Test that client use channels to all running nodes while nodes stop
+ */
+ @Test
+ public void testClientDiscoveryNodesLeave() throws Exception {
+ startGrids(MAX_CLUSTER_SIZE);
+ awaitPartitionMapExchange();
+
+ initClient(getClientConfigurationWithDiscovery(), 0, 1, 2, 3);
+ detectTopologyChange();
+
+ for (int i = MAX_CLUSTER_SIZE - 1; i != 0; i--) {
+ int[] workChannels = IntStream.range(0, i).toArray();
+
+ channels[i] = null;
+ stopGrid(i);
+ awaitPartitionMapExchange();
+ detectTopologyChange();
+
+ awaitChannelsInit(workChannels);
Review comment:
fixied
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r505255508
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -606,16 +603,271 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /**
+ * Should the channel initialization be stopped.
+ */
+ private boolean shouldStopChannelsReinit() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ * @return {@code true} if holders are reinited and {@code false} if the initialization was interrupted.
+ */
+ synchronized boolean initChannelHolders(boolean force) {
+ List<ClientChannelHolder> holders = channels;
+
+ if (!force && holders != null)
+ return true;
+
+ startChannelsReInit = System.currentTimeMillis();
+
+ // Enable parallel threads to schedule new init of channel holders.
+ scheduledChannelsReinit.set(false);
+
+ Map<InetSocketAddress, Integer> newAddrs = null;
+
+ if (clientCfg.getAddressesFinder() != null) {
+ String[] hostAddrs = clientCfg.getAddressesFinder().getAddresses();
+
+ if (hostAddrs.length == 0)
+ throw new ClientException("Empty addresses");
+
+ if (!Arrays.equals(hostAddrs, prevHostAddrs)) {
+ newAddrs = parsedAddresses(hostAddrs);
+ prevHostAddrs = hostAddrs;
+ }
+ } else if (holders == null)
+ newAddrs = parsedAddresses(clientCfg.getAddresses());
+
+ if (newAddrs == null) {
+ finishChannelsReInit = System.currentTimeMillis();
+ return true;
+ }
+
+ Map<InetSocketAddress, ClientChannelHolder> curAddrs = new HashMap<>();
+ Set<InetSocketAddress> allAddrs = new HashSet<>(newAddrs.keySet());
+
+ if (holders != null) {
+ for (int i = 0; i < holders.size(); i++) {
+ ClientChannelHolder h = holders.get(i);
+
+ curAddrs.put(h.chCfg.getAddress(), h);
+ allAddrs.add(h.chCfg.getAddress());
+ }
+ }
+
+ List<ClientChannelHolder> reinitHolders = new ArrayList<>();
+
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currDfltHolder = null;
+
+ int idx = curChIdx;
+
+ if (idx != -1)
+ currDfltHolder = holders.get(idx);
+
+ for (InetSocketAddress addr : allAddrs) {
+ if (shouldStopChannelsReinit())
+ return false;
+
+ // Obsolete addr, to be removed.
+ if (!newAddrs.containsKey(addr)) {
+ curAddrs.get(addr).close();
+
+ continue;
+ }
+
+ // Create new holders for new addrs.
+ if (!curAddrs.containsKey(addr)) {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+
+ for (int i = 0; i < newAddrs.get(addr); i++)
+ reinitHolders.add(hld);
+
+ continue;
+ }
+
+ // This holder is up to date.
+ ClientChannelHolder hld = curAddrs.get(addr);
+
+ for (int i = 0; i < newAddrs.get(addr); i++)
+ reinitHolders.add(hld);
+
+ if (hld == currDfltHolder)
+ dfltChannelIdx = reinitHolders.size() - 1;
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(reinitHolders.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels = reinitHolders;
+ curChIdx = dfltChannelIdx;
+ }
+ finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+
+ finishChannelsReInit = System.currentTimeMillis();
+ return true;
+ }
+
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void channelsInit(boolean force) {
+ if (!force && channels != null)
Review comment:
fixed
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -512,80 +504,81 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
return ranges.stream()
.flatMap(r -> IntStream
.rangeClosed(r.portFrom(), r.portTo()).boxed()
- .map(p -> new InetSocketAddress(r.host(), p))
+ .map(p -> InetSocketAddress.createUnresolved(r.host(), p))
)
- .collect(Collectors.toList());
+ .collect(Collectors.toMap(a -> a, a -> 1, Integer::sum));
}
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
+ /**
+ * Roll current default channel if specified holder equals to it.
+ */
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
+ int idx = curChIdx;
+ List<ClientChannelHolder> holders = channels;
- throw e;
- }
- }
+ ClientChannelHolder dfltHld = holders.get(idx);
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ if (dfltHld == hld) {
+ idx += 1;
+
+ if (idx >= holders.size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On current channel failure.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
+ private void onChannelFailure(ClientChannel ch) {
// There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
// when current index was changed and no other wrong channel will be closed by current thread because
// onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ onChannelFailure(channels.get(curChIdx), ch);
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ chFailLsnrs.forEach(Runnable::run);
+
+ if (scheduledChannelsReinit.get())
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Thin client kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482730378
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +65,16 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
/** Index of the current channel. */
- private int curChIdx;
+ private volatile int curChIdx = -1;
- /** Partition awareness enabled. */
- private final boolean partitionAwarenessEnabled;
+ /** Is all channels should be initialized at one moment. */
Review comment:
`Is` -> `Whether`
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +116,33 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /**
+ * Try use other limited number of channels to send a request if default channel is not responding.
Review comment:
Please move this explanation to public setter/getter
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ClientChannelHolder.java
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.ignite.client.ClientAuthenticationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * Channels holder.
+ */
+class ClientChannelHolder {
+ /** Channel configuration. */
+ final ClientChannelConfiguration chCfg;
+
+ /** Channel. */
+ private volatile ClientChannel ch;
+
+ /** Address that holder is bind to (chCfg.addr) is not in use now. So close the holder */
+ private volatile boolean close;
+
+ /** Timestamps of reconnect retries. */
+ private final long[] reconnectRetries;
+
+ /** Channel factory. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
+
+ /** Callback invokes when new channel create */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate;
+
+ /** Callback invokes when channel close */
+ private final Consumer<ClientChannel> onChannelClose;
+
+ /**
+ * @param chCfg Channel config.
+ */
+ ClientChannelHolder(ClientChannelConfiguration chCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate,
+ Consumer<ClientChannel> onChannelClose) {
+ this.chCfg = chCfg;
+ this.chFactory = chFactory;
+ this.onChannelCreate = onChannelCreate;
+ this.onChannelClose = onChannelClose;
+
+ reconnectRetries = chCfg.getReconnectThrottlingRetries() > 0 && chCfg.getReconnectThrottlingPeriod() > 0L ?
+ new long[chCfg.getReconnectThrottlingRetries()] : null;
+ }
+
+ /**
+ * @return Whether reconnect throttling should be applied.
+ */
+ boolean applyReconnectionThrottling() {
Review comment:
`shouldApplyReconnectionThrottling`
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/kubernetes/KubernetesConnectorConfigurator.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.kubernetes;
+
+/**
+ * Interface provide methods to configure Kubernetes connection.
+ */
+public interface KubernetesConnectorConfigurator {
Review comment:
I think we should remove this interface. There is hardly any value in sharing an interface across thin and thick configurations, but it can cause difficulties when APIs evolve in the future.
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/spi/discovery/tcp/ipfinder/kubernetes/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes;
+
+import java.net.InetAddress;
+import java.util.function.Supplier;
+import org.apache.ignite.kubernetes.KubernetesConnectorConfigurator;
+import org.apache.ignite.kubernetes.KubernetesConnectorDefaults;
+import org.apache.ignite.kubernetes.KubernetesServiceAddressResolver;
+
+/**
+ * Address finder for automatic lookup of Ignite nodes running in Kubernetes environment. All Ignite nodes have to
+ * deployed as Kubernetes pods in order to be found. Applications and Ignite nodes running outside of Kubernetes
+ * will not be able to reach the containerized counterparts.
+ * <p>
+ * The implementation is based on a distinct Kubernetes service that has to be created and should be deployed prior
+ * Ignite nodes startup. The service will maintain a list of all endpoints (internal IP addresses) of all containerized
+ * Ignite pods running so far. The name of the service must be equal to {@link #setServiceName(String)} which is
+ * `ignite` by default.
+ * <p>
+ * As for Ignite pods, it's recommended to label them in such a way that the service will use the label in its selector
Review comment:
This paragraph is also confusing. We don't care how users define their services and use their selectors, why mention that?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(
+ new ClientChannelConfiguration(clientCfg, addr), chFactory, onChannelCreate, onChannelClose);
+
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
- affinityCtx = new ClientCacheAffinityContext(binary);
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
- ClientConnectionException lastEx = null;
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
- for (int i = 0; i < channels.length; i++) {
+ curChannelsGuard.writeLock().lock();
Review comment:
Do we need a lock around AtomicReference usage?
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/spi/discovery/tcp/ipfinder/kubernetes/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes;
+
+import java.net.InetAddress;
+import java.util.function.Supplier;
+import org.apache.ignite.kubernetes.KubernetesConnectorConfigurator;
+import org.apache.ignite.kubernetes.KubernetesConnectorDefaults;
+import org.apache.ignite.kubernetes.KubernetesServiceAddressResolver;
+
+/**
+ * Address finder for automatic lookup of Ignite nodes running in Kubernetes environment. All Ignite nodes have to
+ * deployed as Kubernetes pods in order to be found. Applications and Ignite nodes running outside of Kubernetes
+ * will not be able to reach the containerized counterparts.
+ * <p>
+ * The implementation is based on a distinct Kubernetes service that has to be created and should be deployed prior
+ * Ignite nodes startup. The service will maintain a list of all endpoints (internal IP addresses) of all containerized
Review comment:
I know this was copied from existing `TcpDiscoveryKubernetesIpFinder`, but the text is confusing:
* `prior Ignite nodes startup` - not true, k8s service can be created before or after the pods - does not matter
* `service will maintain a list of all endpoints` - misleading, service is an abstraction over a set of pods, it does not maintain endpoints, and this is k8s implementation details https://kubernetes.io/docs/concepts/services-networking/service/
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/spi/discovery/tcp/ipfinder/kubernetes/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes;
+
+import java.net.InetAddress;
+import java.util.function.Supplier;
+import org.apache.ignite.kubernetes.KubernetesConnectorConfigurator;
+import org.apache.ignite.kubernetes.KubernetesConnectorDefaults;
+import org.apache.ignite.kubernetes.KubernetesServiceAddressResolver;
+
+/**
+ * Address finder for automatic lookup of Ignite nodes running in Kubernetes environment. All Ignite nodes have to
Review comment:
have to **be** deployed
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ClientChannelHolder.java
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.ignite.client.ClientAuthenticationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * Channels holder.
+ */
+class ClientChannelHolder {
+ /** Channel configuration. */
+ final ClientChannelConfiguration chCfg;
Review comment:
`private`
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/kubernetes/KubernetesServiceAddressResolver.java
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.kubernetes;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.URL;
+import java.net.UnknownHostException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.security.SecureRandom;
+import java.security.cert.X509Certificate;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import javax.net.ssl.HostnameVerifier;
+import javax.net.ssl.HttpsURLConnection;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSession;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * The class is responsible to fetch list of IP address for all pods that runs the specified kubernetes service.
+ */
+public class KubernetesServiceAddressResolver {
Review comment:
I don't think this should be in the public package, users won't use this class directly.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannelFacade.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.ignite.IgniteBinary;
+import org.apache.ignite.client.ClientAuthenticationException;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+
+/**
+ * Communication channel with failover and partition awareness.
+ */
+final class ReliableChannelFacade implements AutoCloseable {
Review comment:
Can you please explain the purpose of this class?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
Review comment:
`resolvedAddrs` is a List, this can be inefficient with large number of channels and addresses
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/spi/discovery/tcp/ipfinder/kubernetes/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes;
+
+import java.net.InetAddress;
+import java.util.function.Supplier;
+import org.apache.ignite.kubernetes.KubernetesConnectorConfigurator;
+import org.apache.ignite.kubernetes.KubernetesConnectorDefaults;
+import org.apache.ignite.kubernetes.KubernetesServiceAddressResolver;
+
+/**
+ * Address finder for automatic lookup of Ignite nodes running in Kubernetes environment. All Ignite nodes have to
+ * deployed as Kubernetes pods in order to be found. Applications and Ignite nodes running outside of Kubernetes
+ * will not be able to reach the containerized counterparts.
+ * <p>
+ * The implementation is based on a distinct Kubernetes service that has to be created and should be deployed prior
+ * Ignite nodes startup. The service will maintain a list of all endpoints (internal IP addresses) of all containerized
+ * Ignite pods running so far. The name of the service must be equal to {@link #setServiceName(String)} which is
+ * `ignite` by default.
+ * <p>
+ * As for Ignite pods, it's recommended to label them in such a way that the service will use the label in its selector
+ * configuration excluding endpoints of irrelevant Kubernetes pods running in parallel.
+ * <p>
+ * The address finder, in its turn, will call this service to retrieve Ignite pods IP addresses. The port will be
+ * set with {@link ReliableChannel#parseAddresses(String[])}. Make sure that all Ignite pods occupy a similar
Review comment:
`ReliableChannel` is an implementation detail in the internal package, let's not mention it in the javadoc of the public class.
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/spi/discovery/tcp/ipfinder/kubernetes/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes;
Review comment:
I think this belongs to `org.apache.ignite.client` package
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
Review comment:
Copypasted javadoc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504774568
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
Review comment:
Currently master branch do work approach you describe, and it leads to:
1. introducing additional AtomicInteger variable for attempts counter. Counter is required as it used in separate threads execute `handleServiceAsync` method;
2. code duplication for iterating through channels (for sync and async methods).
This is reasons to use T2. It's not pretty, but the alternative doesn't looks better, I think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504828714
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -512,80 +504,81 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
return ranges.stream()
.flatMap(r -> IntStream
.rangeClosed(r.portFrom(), r.portTo()).boxed()
- .map(p -> new InetSocketAddress(r.host(), p))
+ .map(p -> InetSocketAddress.createUnresolved(r.host(), p))
)
- .collect(Collectors.toList());
+ .collect(Collectors.toMap(a -> a, a -> 1, Integer::sum));
}
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
+ /**
+ * Roll current default channel if specified holder equals to it.
+ */
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
+ int idx = curChIdx;
+ List<ClientChannelHolder> holders = channels;
- throw e;
- }
- }
+ ClientChannelHolder dfltHld = holders.get(idx);
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ if (dfltHld == hld) {
+ idx += 1;
+
+ if (idx >= holders.size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On current channel failure.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
+ private void onChannelFailure(ClientChannel ch) {
// There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
// when current index was changed and no other wrong channel will be closed by current thread because
// onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ onChannelFailure(channels.get(curChIdx), ch);
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ chFailLsnrs.forEach(Runnable::run);
+
+ if (scheduledChannelsReinit.get())
+ channelsInit(true);
+ else
+ rollCurrentChannel(hld);
Review comment:
Also, if new default channel is assigned by `channelsInit()`, than `rollCurrentChannel(hld)` will do no-op, since `hld` already not a default channel.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r493504987
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** */
+ private void checkDoesNotReinit(ClientConfiguration ccfg) {
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ List<ReliableChannel.ClientChannelHolder> originalChannels = rc.getChannelHolders();
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+
+ assertSame(originalChannels, newChannels);
+ IntStream.range(0, 3).forEach(i -> {
+ assertSame(originalChannels.get(i), newChannels.get(i));
Review comment:
At first we check that reference of List doesn't change, then check that reference for every holder within the List doesn't change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r488673289
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -169,8 +167,10 @@
// No-op.
}
- for (ClientChannelHolder hld : channels)
- hld.closeChannel();
+ if (channels.get() != null) {
+ for (ClientChannelHolder hld: channels.get())
Review comment:
There is no race conditions. As channels are set within synchronized method `initChannelHolders`. The discussed code is placed within `close` that is synchronized too. Also channels is never reference to null after it initialized, so this check is enough.
For other accessors to the `channels` field:
1. `rollCurrentChannel` invokes after channels is initialized. Also it guards with `curChannelsGuard.writeLock`, so it is safe the reference directly;
2. `applyOnDefaultChannel` can lead to NPE in case `initConnection` is not invoked. So I've just lazy initialization in the method, so there is a guarantee that channels are initialized. In our case this method call just returns fast;
3. `initAllChannelsAsync` are invoked after channels are initialized. Also it uses `ArrayList.iterator()` to iterate through channels. Under the hood it just creates Iterator object that. We change `channels` by reference switch so, this iterator is safe to use.
4. `channelsInit` is checks channels for null and then initialize them is it needed. So it's safe too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504769724
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
+
+ } catch (Throwable ex) {
+ fut.completeExceptionally(ex);
+ return;
}
- if (err instanceof ClientConnectionException) {
- onChannelFailure(ch);
+ ClientChannel ch = chAndAttempts.getKey();
- if (failure == null)
- failure = (ClientConnectionException) err;
- else
- failure.addSuppressed(err);
+ ch
+ .serviceAsync(op, payloadWriter, payloadReader)
+ .handle((res, err) -> {
+ if (err == null) {
+ fut.complete(res);
Review comment:
No, I mean missing empty line between `fut.complete(res);` and `onChannelFailure(ch);`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482792589
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/kubernetes/KubernetesConnectorConfigurator.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.kubernetes;
+
+/**
+ * Interface provide methods to configure Kubernetes connection.
+ */
+public interface KubernetesConnectorConfigurator {
Review comment:
This interface extends our public API. Every new public API is a future liability and should be considered carefully. In this case there is no value for the users, so let's remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r489276021
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -169,8 +167,10 @@
// No-op.
}
- for (ClientChannelHolder hld : channels)
- hld.closeChannel();
+ if (channels.get() != null) {
+ for (ClientChannelHolder hld: channels.get())
Review comment:
Fixed. The only place channels used directly is a single getter in `applyOnDefaultChannel` within guard read lock.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r486971241
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
Review comment:
Same here - let's revert and move the logic to `ReliableChannel`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
+ this.addrs = Arrays.copyOf(addrs, addrs.length);
+ addrFinder = () -> this.addrs;
+ }
+
+ return this;
+ }
+
+ /**
+ * @param finder function that finds node addresses
+ */
+ public ClientConfiguration setAddressesFinder(Supplier<String[]> finder) {
Review comment:
Corresponding getter is missing.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -169,8 +167,10 @@
// No-op.
}
- for (ClientChannelHolder hld : channels)
- hld.closeChannel();
+ if (channels.get() != null) {
+ for (ClientChannelHolder hld: channels.get())
Review comment:
Here and below: race condition - `channels` may become null after the null check. Copy to a variable, then check for null / iterate / etc.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
/** Index of the current channel. */
- private int curChIdx;
+ private volatile int curChIdx = -1;
/** Partition awareness enabled. */
private final boolean partitionAwarenessEnabled;
/** Cache partition awareness context. */
- private final ClientCacheAffinityContext affinityCtx;
+ private final ClientCacheAffinityContext affCtx;
Review comment:
Please revert unnecessary renames to reduce the scope of changes.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
Is there a reason to bring this outside of the constructor?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
Review comment:
`AtomicReference` is not needed, we can just use `volatile` since only `get()` and `set()` methods are used below.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -493,6 +520,25 @@ public ClientConfiguration setReconnectThrottlingRetries(int reconnectThrottling
return this;
}
+ /**
+ * Get retry limit.
+ */
+ public int getRetryLimit() {
+ return retryLimit;
+ }
+
+ /**
+ * Try use limited number of channels to send a request if default channel is not responding.
Review comment:
```
Sets the retry limit. When a request fails due to a connection error, and multiple server connections are available, Ignite will retry the request on every connection. When this property is greater than zero, Ignite will limit the number of retries.
```
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
Review comment:
Let's revert - no need to be clever here. `ClientConfiguration` should be a simple POJO, the logic for address handling should not be here.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -381,78 +343,59 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
.rangeClosed(r.portFrom(), r.portTo()).boxed()
.map(p -> new InetSocketAddress(r.host(), p))
)
- .collect(Collectors.toList());
- }
-
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
- try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
-
- throw e;
- }
- }
-
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ .collect(Collectors.toSet());
}
/**
- * On current channel failure.
+ * Roll current default channel if specified holder equals to it.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
- // There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
- // when current index was changed and no other wrong channel will be closed by current thread because
- // onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
+ try {
+ ClientChannelHolder dfltHld = channels.get().get(curChIdx);
+ if (dfltHld == hld) {
+ int idx = curChIdx + 1;
+ if (idx >= channels.get().size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (hld != null && ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ rollCurrentChannel(hld);
+
+ chFailLsnrs.forEach(Runnable::run);
}
/**
* Asynchronously try to establish a connection to all configured servers.
*/
- private void initAllChannelsAsync() {
- // Skip if there is already channels reinit scheduled.
- if (scheduledChannelsReinit.compareAndSet(false, true)) {
- asyncRunner.submit(
- () -> {
- scheduledChannelsReinit.set(false);
-
- for (ClientChannelHolder hld : channels) {
- if (scheduledChannelsReinit.get() || closed)
- return; // New reinit task scheduled or channel is closed.
-
- try {
- hld.getOrCreateChannel(true);
- }
- catch (Exception ignore) {
- // No-op.
- }
+ void initAllChannelsAsync() {
Review comment:
`private`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
Review comment:
`shouldStopChannelsReinit()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
Review comment:
This requires iteration over entire map. Instead, use `addrs.entrySet()` for looping and remove by key.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
+ if (!force && channels.get() != null)
+ return;
+
+ // Skip if there is already channels reinit scheduled.
+ // Flag is set back when a thread comes in synchronized initChannelHolders
+ if (scheduledChannelsReinit.compareAndSet(false, true)) {
+ initChannelHolders(force);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = Optional
Review comment:
Too clever, verbose, and memory-hungry. Just make it `channel = hld == null ? null : hld.getOrCreateChannel();`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -33,9 +36,16 @@
/** Serial version uid. */
private static final long serialVersionUID = 0L;
- /** @serial Server addresses. */
+ /**
+ * @serial Server addresses.
+ * @deprecated deprecated in favor of {@link #addrFinder}. Keep it due to Serializable compatibility.
+ */
+ @Deprecated
private String[] addrs = null;
+ /** Server addresses finder. */
+ private transient Supplier<String[]> addrFinder;
Review comment:
I would prefer a dedicated interface - `ThinClientAddressFinder` or something like this.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
Review comment:
Same here - let's revert and move the logic to `ReliableChannel`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
+ this.addrs = Arrays.copyOf(addrs, addrs.length);
+ addrFinder = () -> this.addrs;
+ }
+
+ return this;
+ }
+
+ /**
+ * @param finder function that finds node addresses
+ */
+ public ClientConfiguration setAddressesFinder(Supplier<String[]> finder) {
Review comment:
Corresponding getter is missing.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -169,8 +167,10 @@
// No-op.
}
- for (ClientChannelHolder hld : channels)
- hld.closeChannel();
+ if (channels.get() != null) {
+ for (ClientChannelHolder hld: channels.get())
Review comment:
Here and below: race condition - `channels` may become null after the null check. Copy to a variable, then check for null / iterate / etc.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
/** Index of the current channel. */
- private int curChIdx;
+ private volatile int curChIdx = -1;
/** Partition awareness enabled. */
private final boolean partitionAwarenessEnabled;
/** Cache partition awareness context. */
- private final ClientCacheAffinityContext affinityCtx;
+ private final ClientCacheAffinityContext affCtx;
Review comment:
Please revert unnecessary renames to reduce the scope of changes.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
Is there a reason to bring this outside of the constructor?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
Review comment:
`AtomicReference` is not needed, we can just use `volatile` since only `get()` and `set()` methods are used below.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -493,6 +520,25 @@ public ClientConfiguration setReconnectThrottlingRetries(int reconnectThrottling
return this;
}
+ /**
+ * Get retry limit.
+ */
+ public int getRetryLimit() {
+ return retryLimit;
+ }
+
+ /**
+ * Try use limited number of channels to send a request if default channel is not responding.
Review comment:
```
Sets the retry limit. When a request fails due to a connection error, and multiple server connections are available, Ignite will retry the request on every connection. When this property is greater than zero, Ignite will limit the number of retries.
```
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
Review comment:
Let's revert - no need to be clever here. `ClientConfiguration` should be a simple POJO, the logic for address handling should not be here.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -381,78 +343,59 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
.rangeClosed(r.portFrom(), r.portTo()).boxed()
.map(p -> new InetSocketAddress(r.host(), p))
)
- .collect(Collectors.toList());
- }
-
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
- try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
-
- throw e;
- }
- }
-
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ .collect(Collectors.toSet());
}
/**
- * On current channel failure.
+ * Roll current default channel if specified holder equals to it.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
- // There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
- // when current index was changed and no other wrong channel will be closed by current thread because
- // onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
+ try {
+ ClientChannelHolder dfltHld = channels.get().get(curChIdx);
+ if (dfltHld == hld) {
+ int idx = curChIdx + 1;
+ if (idx >= channels.get().size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (hld != null && ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ rollCurrentChannel(hld);
+
+ chFailLsnrs.forEach(Runnable::run);
}
/**
* Asynchronously try to establish a connection to all configured servers.
*/
- private void initAllChannelsAsync() {
- // Skip if there is already channels reinit scheduled.
- if (scheduledChannelsReinit.compareAndSet(false, true)) {
- asyncRunner.submit(
- () -> {
- scheduledChannelsReinit.set(false);
-
- for (ClientChannelHolder hld : channels) {
- if (scheduledChannelsReinit.get() || closed)
- return; // New reinit task scheduled or channel is closed.
-
- try {
- hld.getOrCreateChannel(true);
- }
- catch (Exception ignore) {
- // No-op.
- }
+ void initAllChannelsAsync() {
Review comment:
`private`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
Review comment:
`shouldStopChannelsReinit()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
Review comment:
This requires iteration over entire map. Instead, use `addrs.entrySet()` for looping and remove by key.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
+ if (!force && channels.get() != null)
+ return;
+
+ // Skip if there is already channels reinit scheduled.
+ // Flag is set back when a thread comes in synchronized initChannelHolders
+ if (scheduledChannelsReinit.compareAndSet(false, true)) {
+ initChannelHolders(force);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = Optional
Review comment:
Too clever, verbose, and memory-hungry. Just make it `channel = hld == null ? null : hld.getOrCreateChannel();`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -33,9 +36,16 @@
/** Serial version uid. */
private static final long serialVersionUID = 0L;
- /** @serial Server addresses. */
+ /**
+ * @serial Server addresses.
+ * @deprecated deprecated in favor of {@link #addrFinder}. Keep it due to Serializable compatibility.
+ */
+ @Deprecated
private String[] addrs = null;
+ /** Server addresses finder. */
+ private transient Supplier<String[]> addrFinder;
Review comment:
I would prefer a dedicated interface - `ThinClientAddressFinder` or something like this.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
Review comment:
Same here - let's revert and move the logic to `ReliableChannel`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
+ this.addrs = Arrays.copyOf(addrs, addrs.length);
+ addrFinder = () -> this.addrs;
+ }
+
+ return this;
+ }
+
+ /**
+ * @param finder function that finds node addresses
+ */
+ public ClientConfiguration setAddressesFinder(Supplier<String[]> finder) {
Review comment:
Corresponding getter is missing.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -169,8 +167,10 @@
// No-op.
}
- for (ClientChannelHolder hld : channels)
- hld.closeChannel();
+ if (channels.get() != null) {
+ for (ClientChannelHolder hld: channels.get())
Review comment:
Here and below: race condition - `channels` may become null after the null check. Copy to a variable, then check for null / iterate / etc.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
/** Index of the current channel. */
- private int curChIdx;
+ private volatile int curChIdx = -1;
/** Partition awareness enabled. */
private final boolean partitionAwarenessEnabled;
/** Cache partition awareness context. */
- private final ClientCacheAffinityContext affinityCtx;
+ private final ClientCacheAffinityContext affCtx;
Review comment:
Please revert unnecessary renames to reduce the scope of changes.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
Is there a reason to bring this outside of the constructor?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
Review comment:
`AtomicReference` is not needed, we can just use `volatile` since only `get()` and `set()` methods are used below.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -493,6 +520,25 @@ public ClientConfiguration setReconnectThrottlingRetries(int reconnectThrottling
return this;
}
+ /**
+ * Get retry limit.
+ */
+ public int getRetryLimit() {
+ return retryLimit;
+ }
+
+ /**
+ * Try use limited number of channels to send a request if default channel is not responding.
Review comment:
```
Sets the retry limit. When a request fails due to a connection error, and multiple server connections are available, Ignite will retry the request on every connection. When this property is greater than zero, Ignite will limit the number of retries.
```
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
Review comment:
Let's revert - no need to be clever here. `ClientConfiguration` should be a simple POJO, the logic for address handling should not be here.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -381,78 +343,59 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
.rangeClosed(r.portFrom(), r.portTo()).boxed()
.map(p -> new InetSocketAddress(r.host(), p))
)
- .collect(Collectors.toList());
- }
-
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
- try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
-
- throw e;
- }
- }
-
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ .collect(Collectors.toSet());
}
/**
- * On current channel failure.
+ * Roll current default channel if specified holder equals to it.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
- // There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
- // when current index was changed and no other wrong channel will be closed by current thread because
- // onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
+ try {
+ ClientChannelHolder dfltHld = channels.get().get(curChIdx);
+ if (dfltHld == hld) {
+ int idx = curChIdx + 1;
+ if (idx >= channels.get().size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (hld != null && ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ rollCurrentChannel(hld);
+
+ chFailLsnrs.forEach(Runnable::run);
}
/**
* Asynchronously try to establish a connection to all configured servers.
*/
- private void initAllChannelsAsync() {
- // Skip if there is already channels reinit scheduled.
- if (scheduledChannelsReinit.compareAndSet(false, true)) {
- asyncRunner.submit(
- () -> {
- scheduledChannelsReinit.set(false);
-
- for (ClientChannelHolder hld : channels) {
- if (scheduledChannelsReinit.get() || closed)
- return; // New reinit task scheduled or channel is closed.
-
- try {
- hld.getOrCreateChannel(true);
- }
- catch (Exception ignore) {
- // No-op.
- }
+ void initAllChannelsAsync() {
Review comment:
`private`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
Review comment:
`shouldStopChannelsReinit()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
Review comment:
This requires iteration over entire map. Instead, use `addrs.entrySet()` for looping and remove by key.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
+ if (!force && channels.get() != null)
+ return;
+
+ // Skip if there is already channels reinit scheduled.
+ // Flag is set back when a thread comes in synchronized initChannelHolders
+ if (scheduledChannelsReinit.compareAndSet(false, true)) {
+ initChannelHolders(force);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = Optional
Review comment:
Too clever, verbose, and memory-hungry. Just make it `channel = hld == null ? null : hld.getOrCreateChannel();`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -33,9 +36,16 @@
/** Serial version uid. */
private static final long serialVersionUID = 0L;
- /** @serial Server addresses. */
+ /**
+ * @serial Server addresses.
+ * @deprecated deprecated in favor of {@link #addrFinder}. Keep it due to Serializable compatibility.
+ */
+ @Deprecated
private String[] addrs = null;
+ /** Server addresses finder. */
+ private transient Supplier<String[]> addrFinder;
Review comment:
I would prefer a dedicated interface - `ThinClientAddressFinder` or something like this.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
Review comment:
Same here - let's revert and move the logic to `ReliableChannel`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
+ this.addrs = Arrays.copyOf(addrs, addrs.length);
+ addrFinder = () -> this.addrs;
+ }
+
+ return this;
+ }
+
+ /**
+ * @param finder function that finds node addresses
+ */
+ public ClientConfiguration setAddressesFinder(Supplier<String[]> finder) {
Review comment:
Corresponding getter is missing.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -169,8 +167,10 @@
// No-op.
}
- for (ClientChannelHolder hld : channels)
- hld.closeChannel();
+ if (channels.get() != null) {
+ for (ClientChannelHolder hld: channels.get())
Review comment:
Here and below: race condition - `channels` may become null after the null check. Copy to a variable, then check for null / iterate / etc.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
/** Index of the current channel. */
- private int curChIdx;
+ private volatile int curChIdx = -1;
/** Partition awareness enabled. */
private final boolean partitionAwarenessEnabled;
/** Cache partition awareness context. */
- private final ClientCacheAffinityContext affinityCtx;
+ private final ClientCacheAffinityContext affCtx;
Review comment:
Please revert unnecessary renames to reduce the scope of changes.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
Is there a reason to bring this outside of the constructor?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
Review comment:
`AtomicReference` is not needed, we can just use `volatile` since only `get()` and `set()` methods are used below.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -493,6 +520,25 @@ public ClientConfiguration setReconnectThrottlingRetries(int reconnectThrottling
return this;
}
+ /**
+ * Get retry limit.
+ */
+ public int getRetryLimit() {
+ return retryLimit;
+ }
+
+ /**
+ * Try use limited number of channels to send a request if default channel is not responding.
Review comment:
```
Sets the retry limit. When a request fails due to a connection error, and multiple server connections are available, Ignite will retry the request on every connection. When this property is greater than zero, Ignite will limit the number of retries.
```
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
Review comment:
Let's revert - no need to be clever here. `ClientConfiguration` should be a simple POJO, the logic for address handling should not be here.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -381,78 +343,59 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
.rangeClosed(r.portFrom(), r.portTo()).boxed()
.map(p -> new InetSocketAddress(r.host(), p))
)
- .collect(Collectors.toList());
- }
-
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
- try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
-
- throw e;
- }
- }
-
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ .collect(Collectors.toSet());
}
/**
- * On current channel failure.
+ * Roll current default channel if specified holder equals to it.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
- // There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
- // when current index was changed and no other wrong channel will be closed by current thread because
- // onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
+ try {
+ ClientChannelHolder dfltHld = channels.get().get(curChIdx);
+ if (dfltHld == hld) {
+ int idx = curChIdx + 1;
+ if (idx >= channels.get().size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (hld != null && ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ rollCurrentChannel(hld);
+
+ chFailLsnrs.forEach(Runnable::run);
}
/**
* Asynchronously try to establish a connection to all configured servers.
*/
- private void initAllChannelsAsync() {
- // Skip if there is already channels reinit scheduled.
- if (scheduledChannelsReinit.compareAndSet(false, true)) {
- asyncRunner.submit(
- () -> {
- scheduledChannelsReinit.set(false);
-
- for (ClientChannelHolder hld : channels) {
- if (scheduledChannelsReinit.get() || closed)
- return; // New reinit task scheduled or channel is closed.
-
- try {
- hld.getOrCreateChannel(true);
- }
- catch (Exception ignore) {
- // No-op.
- }
+ void initAllChannelsAsync() {
Review comment:
`private`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
Review comment:
`shouldStopChannelsReinit()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
Review comment:
This requires iteration over entire map. Instead, use `addrs.entrySet()` for looping and remove by key.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
+ if (!force && channels.get() != null)
+ return;
+
+ // Skip if there is already channels reinit scheduled.
+ // Flag is set back when a thread comes in synchronized initChannelHolders
+ if (scheduledChannelsReinit.compareAndSet(false, true)) {
+ initChannelHolders(force);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = Optional
Review comment:
Too clever, verbose, and memory-hungry. Just make it `channel = hld == null ? null : hld.getOrCreateChannel();`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -33,9 +36,16 @@
/** Serial version uid. */
private static final long serialVersionUID = 0L;
- /** @serial Server addresses. */
+ /**
+ * @serial Server addresses.
+ * @deprecated deprecated in favor of {@link #addrFinder}. Keep it due to Serializable compatibility.
+ */
+ @Deprecated
private String[] addrs = null;
+ /** Server addresses finder. */
+ private transient Supplier<String[]> addrFinder;
Review comment:
I would prefer a dedicated interface - `ThinClientAddressFinder` or something like this.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
Review comment:
Same here - let's revert and move the logic to `ReliableChannel`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
+ .map(Supplier::get)
+ .orElse(addrs);
}
/**
* @param addrs Host addresses.
*/
public ClientConfiguration setAddresses(String... addrs) {
- this.addrs = addrs;
+ if (addrs != null) {
+ this.addrs = Arrays.copyOf(addrs, addrs.length);
+ addrFinder = () -> this.addrs;
+ }
+
+ return this;
+ }
+
+ /**
+ * @param finder function that finds node addresses
+ */
+ public ClientConfiguration setAddressesFinder(Supplier<String[]> finder) {
Review comment:
Corresponding getter is missing.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -169,8 +167,10 @@
// No-op.
}
- for (ClientChannelHolder hld : channels)
- hld.closeChannel();
+ if (channels.get() != null) {
+ for (ClientChannelHolder hld: channels.get())
Review comment:
Here and below: race condition - `channels` may become null after the null check. Copy to a variable, then check for null / iterate / etc.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
/** Index of the current channel. */
- private int curChIdx;
+ private volatile int curChIdx = -1;
/** Partition awareness enabled. */
private final boolean partitionAwarenessEnabled;
/** Cache partition awareness context. */
- private final ClientCacheAffinityContext affinityCtx;
+ private final ClientCacheAffinityContext affCtx;
Review comment:
Please revert unnecessary renames to reduce the scope of changes.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpIgniteClient.java
##########
@@ -114,6 +114,7 @@ private TcpIgniteClient(ClientConfiguration cfg) throws ClientException {
binary = new ClientBinary(marsh);
ch = new ReliableChannel(chFactory, cfg, binary);
+ ch.initConnection();
Review comment:
Is there a reason to bring this outside of the constructor?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
Review comment:
`AtomicReference` is not needed, we can just use `volatile` since only `get()` and `set()` methods are used below.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -493,6 +520,25 @@ public ClientConfiguration setReconnectThrottlingRetries(int reconnectThrottling
return this;
}
+ /**
+ * Get retry limit.
+ */
+ public int getRetryLimit() {
+ return retryLimit;
+ }
+
+ /**
+ * Try use limited number of channels to send a request if default channel is not responding.
Review comment:
```
Sets the retry limit. When a request fails due to a connection error, and multiple server connections are available, Ignite will retry the request on every connection. When this property is greater than zero, Ignite will limit the number of retries.
```
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +125,35 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return Optional.ofNullable(addrFinder)
Review comment:
Let's revert - no need to be clever here. `ClientConfiguration` should be a simple POJO, the logic for address handling should not be here.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -381,78 +343,59 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
.rangeClosed(r.portFrom(), r.portTo()).boxed()
.map(p -> new InetSocketAddress(r.host(), p))
)
- .collect(Collectors.toList());
- }
-
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
- try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
-
- throw e;
- }
- }
-
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ .collect(Collectors.toSet());
}
/**
- * On current channel failure.
+ * Roll current default channel if specified holder equals to it.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
- // There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
- // when current index was changed and no other wrong channel will be closed by current thread because
- // onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
+ try {
+ ClientChannelHolder dfltHld = channels.get().get(curChIdx);
+ if (dfltHld == hld) {
+ int idx = curChIdx + 1;
+ if (idx >= channels.get().size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (hld != null && ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ rollCurrentChannel(hld);
+
+ chFailLsnrs.forEach(Runnable::run);
}
/**
* Asynchronously try to establish a connection to all configured servers.
*/
- private void initAllChannelsAsync() {
- // Skip if there is already channels reinit scheduled.
- if (scheduledChannelsReinit.compareAndSet(false, true)) {
- asyncRunner.submit(
- () -> {
- scheduledChannelsReinit.set(false);
-
- for (ClientChannelHolder hld : channels) {
- if (scheduledChannelsReinit.get() || closed)
- return; // New reinit task scheduled or channel is closed.
-
- try {
- hld.getOrCreateChannel(true);
- }
- catch (Exception ignore) {
- // No-op.
- }
+ void initAllChannelsAsync() {
Review comment:
`private`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
Review comment:
`shouldStopChannelsReinit()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
Review comment:
This requires iteration over entire map. Instead, use `addrs.entrySet()` for looping and remove by key.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
+ if (!force && channels.get() != null)
+ return;
+
+ // Skip if there is already channels reinit scheduled.
+ // Flag is set back when a thread comes in synchronized initChannelHolders
+ if (scheduledChannelsReinit.compareAndSet(false, true)) {
+ initChannelHolders(force);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = Optional
Review comment:
Too clever, verbose, and memory-hungry. Just make it `channel = hld == null ? null : hld.getOrCreateChannel();`.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -33,9 +36,16 @@
/** Serial version uid. */
private static final long serialVersionUID = 0L;
- /** @serial Server addresses. */
+ /**
+ * @serial Server addresses.
+ * @deprecated deprecated in favor of {@link #addrFinder}. Keep it due to Serializable compatibility.
+ */
+ @Deprecated
private String[] addrs = null;
+ /** Server addresses finder. */
+ private transient Supplier<String[]> addrFinder;
Review comment:
I would prefer a dedicated interface - `ThinClientAddressFinder` or something like this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504799624
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -512,80 +504,81 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
return ranges.stream()
.flatMap(r -> IntStream
.rangeClosed(r.portFrom(), r.portTo()).boxed()
- .map(p -> new InetSocketAddress(r.host(), p))
+ .map(p -> InetSocketAddress.createUnresolved(r.host(), p))
)
- .collect(Collectors.toList());
+ .collect(Collectors.toMap(a -> a, a -> 1, Integer::sum));
}
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
+ /**
+ * Roll current default channel if specified holder equals to it.
+ */
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
+ int idx = curChIdx;
+ List<ClientChannelHolder> holders = channels;
- throw e;
- }
- }
+ ClientChannelHolder dfltHld = holders.get(idx);
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ if (dfltHld == hld) {
+ idx += 1;
+
+ if (idx >= holders.size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On current channel failure.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
+ private void onChannelFailure(ClientChannel ch) {
// There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
// when current index was changed and no other wrong channel will be closed by current thread because
// onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ onChannelFailure(channels.get(curChIdx), ch);
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ chFailLsnrs.forEach(Runnable::run);
+
+ if (scheduledChannelsReinit.get())
Review comment:
Yeah, it is a correct optimization. But I will add this condition inside if block to avoid additional rolling of current channel.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r494268992
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -461,9 +418,8 @@ private void initAllChannelsAsync() {
* @param ch Channel.
*/
private void onTopologyChanged(ClientChannel ch) {
- if (partitionAwarenessEnabled && affinityCtx.updateLastTopologyVersion(ch.serverTopologyVersion(),
- ch.serverNodeId()))
- initAllChannelsAsync();
+ if (affinityCtx.updateLastTopologyVersion(ch.serverTopologyVersion(), ch.serverNodeId()))
+ channelsInit(true);
Review comment:
If partition awareness is disabled and a new node is joined it's fine to work with the current node until failure (we don't need to find new nodes unless we disconnected from the current).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r497489616
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
+ */
+ @Test
+ public void testClientDiscoveryNodesJoin() throws Exception {
+ for (int i = 0; i < MAX_CLUSTER_SIZE; ++i) {
+ startGrid(i);
+ awaitPartitionMapExchange();
+
+ int[] workChannels = IntStream.rangeClosed(0, i).toArray();
+
+ if (i == 0)
+ initClient(getClientConfigurationWithDiscovery(), workChannels);
+ else {
+ detectTopologyChange();
+ awaitChannelsInit(workChannels);
+ }
+
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Test that client use channels to all running nodes while nodes stop
Review comment:
fixed.
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
Review comment:
fixed.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,16 +429,230 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean shouldStopChannelsReinit() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ synchronized void initChannelHolders(boolean force) {
+ List<ClientChannelHolder> holders = channels;
+
+ if (!force && holders != null)
+ return;
+
+ startChannelsReInit = System.currentTimeMillis();
+
+ // Enable parallel threads to schedule new init of channel holders.
+ scheduledChannelsReinit.set(false);
+
+ Set<InetSocketAddress> newAddrs = null;
+
+ if (clientCfg.getAddressesFinder() != null) {
+ String[] hostAddrs = clientCfg.getAddressesFinder().getServerAddresses();
+
+ if (hostAddrs.length == 0)
+ throw new ClientException("Empty addresses");
+
+ if (!Arrays.equals(hostAddrs, prevHostAddrs)) {
+ newAddrs = parsedAddresses(hostAddrs);
+ prevHostAddrs = hostAddrs;
+ }
+ } else if (holders == null)
+ newAddrs = parsedAddresses(clientCfg.getAddresses());
+
+ if (newAddrs == null) {
+ finishChannelsReInit = System.currentTimeMillis();
+ return;
+ }
+
+ Map<InetSocketAddress, ClientChannelHolder> curAddrs = Collections.emptyMap();
+ Set<InetSocketAddress> allAddrs = new HashSet<>(newAddrs);
+
+ if (holders != null) {
+ curAddrs = holders.stream()
+ .collect(Collectors.toMap(h -> h.chCfg.getAddress(), h -> h));
+
+ allAddrs.addAll(curAddrs.keySet());
+ }
+
+ List<ClientChannelHolder> reinitHolders = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currDfltHolder = null;
+ int idx = curChIdx;
+ if (idx != -1)
+ currDfltHolder = holders.get(idx);
+
+ for (InetSocketAddress addr : allAddrs) {
+ if (shouldStopChannelsReinit())
+ return;
+
+ // Obsolete addr, to be removed.
+ if (!newAddrs.contains(addr)) {
+ curAddrs.get(addr).close();
+
+ continue;
+ }
+
+ // Create new holders for new addrs.
+ if (!curAddrs.containsKey(addr)) {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ reinitHolders.add(hld);
+
+ continue;
+ }
+
+ // This holder is up to date.
+ ClientChannelHolder hld = curAddrs.get(addr);
+ reinitHolders.add(hld);
+ if (hld == currDfltHolder)
+ dfltChannelIdx = reinitHolders.size() - 1;
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(reinitHolders.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels = reinitHolders;
+ curChIdx = dfltChannelIdx;
+ }
+ finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+
+ finishChannelsReInit = System.currentTimeMillis();
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
+ if (!force && channels != null)
+ return;
+
+ // Skip if there is already channels reinit scheduled.
+ // Flag is set back when a thread comes in synchronized initChannelHolders.
+ if (scheduledChannelsReinit.compareAndSet(false, true)) {
+ initChannelHolders(force);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = hld != null ? hld.getOrCreateChannel() : null;
+
+ if (channel != null)
+ return function.apply(channel);
+
+ } catch (ClientConnectionException e) {
+ onChannelFailure(hld, channel);
+ }
+
+ return null;
+ }
+
+ /**
+ * Apply specified {@code function} on any of available channel.
+ */
+ private <T> T applyOnDefaultChannel(Function<ClientChannel, T> function) {
+ List<ClientChannelHolder> holders = channels;
+
+ if (holders == null)
+ throw new ClientException("Connections to nodes aren't initialized.");
+
+ int size = holders.size();
+
+ int attemptsLimit = clientCfg.getRetryLimit() > 0 ?
+ Math.min(clientCfg.getRetryLimit(), size) : size;
+
+ ClientConnectionException failure = null;
+
+ for (int attempt = 0; attempt < attemptsLimit; attempt++) {
+ ClientChannelHolder hld = null;
+ ClientChannel c = null;
+ try {
+ if (closed)
+ throw new ClientException("Channel is closed");
+
+ curChannelsGuard.readLock().lock();
+ try {
+ hld = channels.get(curChIdx);
+ } finally {
+ curChannelsGuard.readLock().unlock();
+ }
+
+ c = hld.getOrCreateChannel();
+ if (c != null)
+ return function.apply(c);
+ }
+ catch (ClientConnectionException e) {
+ if (failure == null)
+ failure = e;
+ else
+ failure.addSuppressed(e);
+
+ onChannelFailure(hld, c);
+ }
+ }
+
+ throw failure;
+ }
+
+ /**
+ * Try apply specified {@code function} on a channel corresponding to {@code tryNodeId}.
+ * If failed then apply the function on any available channel.
+ */
+ private <T> T apply(UUID tryNodeId, Function<ClientChannel, T> function) {
Review comment:
fixed.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -519,29 +689,44 @@ private boolean applyReconnectionThrottling() {
/**
* Get or create channel.
*/
- private synchronized ClientChannel getOrCreateChannel()
+ private ClientChannel getOrCreateChannel()
throws ClientConnectionException, ClientAuthenticationException, ClientProtocolError {
return getOrCreateChannel(false);
}
/**
* Get or create channel.
*/
- private synchronized ClientChannel getOrCreateChannel(boolean ignoreThrottling)
+ private ClientChannel getOrCreateChannel(boolean ignoreThrottling)
throws ClientConnectionException, ClientAuthenticationException, ClientProtocolError {
- if (ch == null) {
- if (!ignoreThrottling && applyReconnectionThrottling())
- throw new ClientConnectionException("Reconnect is not allowed due to applied throttling");
+ if (ch == null && !close) {
+ synchronized (this) {
+ if (close)
+ return null;
+
+ if (ch != null)
+ return ch;
- ch = chFactory.apply(chCfg);
+ if (!ignoreThrottling && applyReconnectionThrottling())
+ throw new ClientConnectionException("Reconnect is not allowed due to applied throttling");
- if (ch.serverNodeId() != null) {
- ch.addTopologyChangeListener(ReliableChannel.this::onTopologyChanged);
- ch.addNotificationListener(ReliableChannel.this);
+ ClientChannel channel = chFactory.apply(chCfg);
- nodeChannels.values().remove(this);
+ if (channel.serverNodeId() != null) {
+ channel.addTopologyChangeListener(ReliableChannel.this::onTopologyChanged);
+ channel.addNotificationListener(ReliableChannel.this);
- nodeChannels.putIfAbsent(ch.serverNodeId(), this);
+ if (serverNodeId == null)
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r501250135
##########
File path: modules/core/src/main/java/org/apache/ignite/client/ClientAddressFinder.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.client;
+
+/**
+ * This interface provides a list of addresses of Ignite server nodes within a cluster. Thin client uses the list to
+ * route user requests. There are cases when the list is not static, for example in cloud environment. In such cases
+ * addresses of nodes and/or number of server nodes can change. Implementation of this interface should handle these.
+ *
+ * Ignite waites for a topology change to trigger {@link #getAddresses()}. There are two modes of how soon Ignite calls
Review comment:
Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504452829
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -118,21 +123,47 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/** Executor for async operations continuations. */
private Executor asyncContinuationExecutor;
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ if (addrs != null)
+ return Arrays.copyOf(addrs, addrs.length);
Review comment:
NL
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
+
+ } catch (Throwable ex) {
+ fut.completeExceptionally(ex);
Review comment:
NL
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
+
Review comment:
Redundant NL
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -606,16 +603,271 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /**
+ * Should the channel initialization be stopped.
+ */
+ private boolean shouldStopChannelsReinit() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ * @return {@code true} if holders are reinited and {@code false} if the initialization was interrupted.
+ */
+ synchronized boolean initChannelHolders(boolean force) {
+ List<ClientChannelHolder> holders = channels;
+
+ if (!force && holders != null)
+ return true;
+
+ startChannelsReInit = System.currentTimeMillis();
+
+ // Enable parallel threads to schedule new init of channel holders.
+ scheduledChannelsReinit.set(false);
+
+ Map<InetSocketAddress, Integer> newAddrs = null;
+
+ if (clientCfg.getAddressesFinder() != null) {
+ String[] hostAddrs = clientCfg.getAddressesFinder().getAddresses();
+
+ if (hostAddrs.length == 0)
+ throw new ClientException("Empty addresses");
+
+ if (!Arrays.equals(hostAddrs, prevHostAddrs)) {
+ newAddrs = parsedAddresses(hostAddrs);
+ prevHostAddrs = hostAddrs;
+ }
+ } else if (holders == null)
+ newAddrs = parsedAddresses(clientCfg.getAddresses());
+
+ if (newAddrs == null) {
+ finishChannelsReInit = System.currentTimeMillis();
+ return true;
+ }
+
+ Map<InetSocketAddress, ClientChannelHolder> curAddrs = new HashMap<>();
+ Set<InetSocketAddress> allAddrs = new HashSet<>(newAddrs.keySet());
+
+ if (holders != null) {
+ for (int i = 0; i < holders.size(); i++) {
+ ClientChannelHolder h = holders.get(i);
+
+ curAddrs.put(h.chCfg.getAddress(), h);
+ allAddrs.add(h.chCfg.getAddress());
+ }
+ }
+
+ List<ClientChannelHolder> reinitHolders = new ArrayList<>();
+
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currDfltHolder = null;
+
+ int idx = curChIdx;
+
+ if (idx != -1)
+ currDfltHolder = holders.get(idx);
+
+ for (InetSocketAddress addr : allAddrs) {
+ if (shouldStopChannelsReinit())
+ return false;
+
+ // Obsolete addr, to be removed.
+ if (!newAddrs.containsKey(addr)) {
+ curAddrs.get(addr).close();
+
+ continue;
+ }
+
+ // Create new holders for new addrs.
+ if (!curAddrs.containsKey(addr)) {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+
+ for (int i = 0; i < newAddrs.get(addr); i++)
+ reinitHolders.add(hld);
+
+ continue;
+ }
+
+ // This holder is up to date.
+ ClientChannelHolder hld = curAddrs.get(addr);
+
+ for (int i = 0; i < newAddrs.get(addr); i++)
+ reinitHolders.add(hld);
+
+ if (hld == currDfltHolder)
+ dfltChannelIdx = reinitHolders.size() - 1;
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(reinitHolders.size());
+
+ curChannelsGuard.writeLock().lock();
Review comment:
NL
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
+
+ } catch (Throwable ex) {
+ fut.completeExceptionally(ex);
+ return;
}
- if (err instanceof ClientConnectionException) {
- onChannelFailure(ch);
+ ClientChannel ch = chAndAttempts.getKey();
- if (failure == null)
- failure = (ClientConnectionException) err;
- else
- failure.addSuppressed(err);
+ ch
+ .serviceAsync(op, payloadWriter, payloadReader)
+ .handle((res, err) -> {
+ if (err == null) {
+ fut.complete(res);
- if (chIdx == null)
- chIdx = new AtomicInteger();
+ return null;
+ }
- while (chIdx.incrementAndGet() < channels.length) {
- try {
- ch = channel();
+ ClientConnectionException failure0 = failure;
+
+ if (err instanceof ClientConnectionException) {
+ try {
+ // Will try to reinit channels if topology changed.
+ onChannelFailure(ch);
+ }
+ catch (Throwable ex) {
+ fut.completeExceptionally(ex);
+ return null;
+ }
- ClientConnectionException failure0 = failure;
- AtomicInteger chIdx0 = chIdx;
- ClientChannel ch0 = ch;
+ if (failure0 == null)
+ failure0 = (ClientConnectionException)err;
+ else
+ failure0.addSuppressed(err);
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res2, err2) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, failure0, chIdx0, ch0, res2, err2));
+ int leftAttempts = attemptsLimit - chAndAttempts.getValue();
+ // If it is a first retry then reset attempts (as for initialization we use only 1 attempt).
+ if (failure == null)
+ leftAttempts = getRetryLimit() - 1;
+
+ if (leftAttempts > 0) {
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, leftAttempts, failure0);
+ return null;
+ }
+ }
+ else {
+ fut.completeExceptionally(err instanceof ClientException ? err : new ClientException(err));
return null;
- } catch (ClientConnectionException e) {
- onChannelFailure(ch);
- failure.addSuppressed(e);
}
- }
- }
- if (failure != null)
- fut.completeExceptionally(failure);
- else
- fut.completeExceptionally(err instanceof ClientException ? err : new ClientException(err));
-
- return null;
+ fut.completeExceptionally(failure0);
Review comment:
NL
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -606,16 +603,271 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /**
+ * Should the channel initialization be stopped.
+ */
+ private boolean shouldStopChannelsReinit() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ * @return {@code true} if holders are reinited and {@code false} if the initialization was interrupted.
+ */
+ synchronized boolean initChannelHolders(boolean force) {
+ List<ClientChannelHolder> holders = channels;
+
+ if (!force && holders != null)
+ return true;
+
+ startChannelsReInit = System.currentTimeMillis();
+
+ // Enable parallel threads to schedule new init of channel holders.
+ scheduledChannelsReinit.set(false);
+
+ Map<InetSocketAddress, Integer> newAddrs = null;
+
+ if (clientCfg.getAddressesFinder() != null) {
+ String[] hostAddrs = clientCfg.getAddressesFinder().getAddresses();
+
+ if (hostAddrs.length == 0)
+ throw new ClientException("Empty addresses");
+
+ if (!Arrays.equals(hostAddrs, prevHostAddrs)) {
+ newAddrs = parsedAddresses(hostAddrs);
+ prevHostAddrs = hostAddrs;
+ }
+ } else if (holders == null)
+ newAddrs = parsedAddresses(clientCfg.getAddresses());
+
+ if (newAddrs == null) {
+ finishChannelsReInit = System.currentTimeMillis();
+ return true;
+ }
+
+ Map<InetSocketAddress, ClientChannelHolder> curAddrs = new HashMap<>();
+ Set<InetSocketAddress> allAddrs = new HashSet<>(newAddrs.keySet());
+
+ if (holders != null) {
+ for (int i = 0; i < holders.size(); i++) {
+ ClientChannelHolder h = holders.get(i);
+
+ curAddrs.put(h.chCfg.getAddress(), h);
+ allAddrs.add(h.chCfg.getAddress());
+ }
+ }
+
+ List<ClientChannelHolder> reinitHolders = new ArrayList<>();
+
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currDfltHolder = null;
+
+ int idx = curChIdx;
+
+ if (idx != -1)
+ currDfltHolder = holders.get(idx);
+
+ for (InetSocketAddress addr : allAddrs) {
+ if (shouldStopChannelsReinit())
+ return false;
+
+ // Obsolete addr, to be removed.
+ if (!newAddrs.containsKey(addr)) {
+ curAddrs.get(addr).close();
+
+ continue;
+ }
+
+ // Create new holders for new addrs.
+ if (!curAddrs.containsKey(addr)) {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+
+ for (int i = 0; i < newAddrs.get(addr); i++)
+ reinitHolders.add(hld);
+
+ continue;
+ }
+
+ // This holder is up to date.
+ ClientChannelHolder hld = curAddrs.get(addr);
+
+ for (int i = 0; i < newAddrs.get(addr); i++)
+ reinitHolders.add(hld);
+
+ if (hld == currDfltHolder)
+ dfltChannelIdx = reinitHolders.size() - 1;
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(reinitHolders.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels = reinitHolders;
+ curChIdx = dfltChannelIdx;
+ }
+ finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+
+ finishChannelsReInit = System.currentTimeMillis();
+ return true;
+ }
+
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void channelsInit(boolean force) {
+ if (!force && channels != null)
+ return;
+
+ // Do not establish connections if interrupted.
+ if (!initChannelHolders(force))
+ return;
+
+ // Apply no-op function. Establish default channel connection.
+ applyOnDefaultChannel(channel -> null);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = hld != null ? hld.getOrCreateChannel() : null;
+
+ if (channel != null)
+ return function.apply(channel);
+
+ } catch (ClientConnectionException e) {
+ onChannelFailure(hld, channel);
+ }
+
+ return null;
+ }
+
+ /** */
+ private <T> T applyOnDefaultChannel(Function<ClientChannel, T> function) {
+ List<ClientChannelHolder> holders = channels;
+
+ if (holders == null)
+ throw new ClientException("Connections to nodes aren't initialized.");
+
+ int size = holders.size();
+
+ int attemptsLimit = clientCfg.getRetryLimit() > 0 ?
+ Math.min(clientCfg.getRetryLimit(), size) : size;
+
+ return applyOnDefaultChannel(function, attemptsLimit).getKey();
+ }
+
+ /**
+ * Apply specified {@code function} on any of available channel.
+ */
+ private <T> T2<T, Integer> applyOnDefaultChannel(Function<ClientChannel, T> function, int attemptsLimit) {
+ ClientConnectionException failure = null;
+
+ for (int attempt = 0; attempt < attemptsLimit; attempt++) {
+ ClientChannelHolder hld = null;
+ ClientChannel c = null;
+
+ try {
+ if (closed)
+ throw new ClientException("Channel is closed");
+
+ curChannelsGuard.readLock().lock();
+
+ try {
+ hld = channels.get(curChIdx);
+ } finally {
+ curChannelsGuard.readLock().unlock();
+ }
+
+ c = hld.getOrCreateChannel();
+
+ if (c != null)
+ return new T2<>(function.apply(c), attempt + 1);
+ }
+ catch (ClientConnectionException e) {
+ if (failure == null)
+ failure = e;
+ else
+ failure.addSuppressed(e);
+
+ onChannelFailure(hld, c);
+ }
+ }
+
+ throw failure;
+ }
+
+ /**
+ * Try apply specified {@code function} on a channel corresponding to {@code tryNodeId}.
+ * If failed then apply the function on any available channel.
+ */
+ private <T> T applyOnNodeChannelWithFallback(UUID tryNodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = nodeChannels.get(tryNodeId);
+
+ int retryLimit = getRetryLimit();
+
+ if (hld != null) {
+ ClientChannel channel = null;
+
+ try {
+ channel = hld.getOrCreateChannel();
+
+ if (channel != null)
+ return function.apply(channel);
+
+ } catch (ClientConnectionException e) {
+ onChannelFailure(hld, channel);
+
+ retryLimit -= 1;
+
+ if (retryLimit == 0)
+ throw e;
+ }
+ }
+
+ return applyOnDefaultChannel(function, retryLimit).getKey();
+ }
+
+ /** Get retry limit. */
+ private int getRetryLimit() {
+ int size = channels.size();
+
+ return clientCfg.getRetryLimit() > 0 ? Math.min(clientCfg.getRetryLimit(), size) : size;
+ }
+
/**
* Channels holder.
*/
- private class ClientChannelHolder {
+ class ClientChannelHolder {
/** Channel configuration. */
private final ClientChannelConfiguration chCfg;
/** Channel. */
private volatile ClientChannel ch;
+ /** ID of the last server node that {@link ch} is or was connected to. */
+ private volatile UUID serverNodeId;
+
+ /** Address that holder is bind to (chCfg.addr) is not in use now. So close the holder */
Review comment:
point at the end
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -561,5 +746,24 @@ private synchronized void closeChannel() {
ch = null;
}
}
+
+ /** Close holder. */
+ void close() {
+ close = true;
+ if (serverNodeId != null)
+ nodeChannels.remove(serverNodeId);
Review comment:
Still not fixed
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -606,16 +603,271 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /**
+ * Should the channel initialization be stopped.
+ */
+ private boolean shouldStopChannelsReinit() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ * @return {@code true} if holders are reinited and {@code false} if the initialization was interrupted.
+ */
+ synchronized boolean initChannelHolders(boolean force) {
+ List<ClientChannelHolder> holders = channels;
+
+ if (!force && holders != null)
+ return true;
+
+ startChannelsReInit = System.currentTimeMillis();
+
+ // Enable parallel threads to schedule new init of channel holders.
+ scheduledChannelsReinit.set(false);
+
+ Map<InetSocketAddress, Integer> newAddrs = null;
+
+ if (clientCfg.getAddressesFinder() != null) {
+ String[] hostAddrs = clientCfg.getAddressesFinder().getAddresses();
+
+ if (hostAddrs.length == 0)
+ throw new ClientException("Empty addresses");
+
+ if (!Arrays.equals(hostAddrs, prevHostAddrs)) {
+ newAddrs = parsedAddresses(hostAddrs);
+ prevHostAddrs = hostAddrs;
+ }
+ } else if (holders == null)
+ newAddrs = parsedAddresses(clientCfg.getAddresses());
+
+ if (newAddrs == null) {
+ finishChannelsReInit = System.currentTimeMillis();
+ return true;
+ }
+
+ Map<InetSocketAddress, ClientChannelHolder> curAddrs = new HashMap<>();
+ Set<InetSocketAddress> allAddrs = new HashSet<>(newAddrs.keySet());
+
+ if (holders != null) {
+ for (int i = 0; i < holders.size(); i++) {
+ ClientChannelHolder h = holders.get(i);
+
+ curAddrs.put(h.chCfg.getAddress(), h);
+ allAddrs.add(h.chCfg.getAddress());
+ }
+ }
+
+ List<ClientChannelHolder> reinitHolders = new ArrayList<>();
+
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currDfltHolder = null;
+
+ int idx = curChIdx;
+
+ if (idx != -1)
+ currDfltHolder = holders.get(idx);
+
+ for (InetSocketAddress addr : allAddrs) {
+ if (shouldStopChannelsReinit())
+ return false;
+
+ // Obsolete addr, to be removed.
+ if (!newAddrs.containsKey(addr)) {
+ curAddrs.get(addr).close();
+
+ continue;
+ }
+
+ // Create new holders for new addrs.
+ if (!curAddrs.containsKey(addr)) {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+
+ for (int i = 0; i < newAddrs.get(addr); i++)
+ reinitHolders.add(hld);
+
+ continue;
+ }
+
+ // This holder is up to date.
+ ClientChannelHolder hld = curAddrs.get(addr);
+
+ for (int i = 0; i < newAddrs.get(addr); i++)
+ reinitHolders.add(hld);
+
+ if (hld == currDfltHolder)
+ dfltChannelIdx = reinitHolders.size() - 1;
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(reinitHolders.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels = reinitHolders;
+ curChIdx = dfltChannelIdx;
+ }
+ finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+
+ finishChannelsReInit = System.currentTimeMillis();
+ return true;
+ }
+
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void channelsInit(boolean force) {
+ if (!force && channels != null)
+ return;
+
+ // Do not establish connections if interrupted.
+ if (!initChannelHolders(force))
+ return;
+
+ // Apply no-op function. Establish default channel connection.
+ applyOnDefaultChannel(channel -> null);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = hld != null ? hld.getOrCreateChannel() : null;
+
+ if (channel != null)
+ return function.apply(channel);
+
+ } catch (ClientConnectionException e) {
+ onChannelFailure(hld, channel);
+ }
+
+ return null;
+ }
+
+ /** */
+ private <T> T applyOnDefaultChannel(Function<ClientChannel, T> function) {
+ List<ClientChannelHolder> holders = channels;
+
+ if (holders == null)
+ throw new ClientException("Connections to nodes aren't initialized.");
+
+ int size = holders.size();
+
+ int attemptsLimit = clientCfg.getRetryLimit() > 0 ?
+ Math.min(clientCfg.getRetryLimit(), size) : size;
Review comment:
Can be replaced with `getRetryLimit()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -606,16 +603,271 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /**
+ * Should the channel initialization be stopped.
+ */
+ private boolean shouldStopChannelsReinit() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ * @return {@code true} if holders are reinited and {@code false} if the initialization was interrupted.
+ */
+ synchronized boolean initChannelHolders(boolean force) {
+ List<ClientChannelHolder> holders = channels;
+
+ if (!force && holders != null)
+ return true;
+
+ startChannelsReInit = System.currentTimeMillis();
+
+ // Enable parallel threads to schedule new init of channel holders.
+ scheduledChannelsReinit.set(false);
+
+ Map<InetSocketAddress, Integer> newAddrs = null;
+
+ if (clientCfg.getAddressesFinder() != null) {
+ String[] hostAddrs = clientCfg.getAddressesFinder().getAddresses();
+
+ if (hostAddrs.length == 0)
+ throw new ClientException("Empty addresses");
+
+ if (!Arrays.equals(hostAddrs, prevHostAddrs)) {
+ newAddrs = parsedAddresses(hostAddrs);
+ prevHostAddrs = hostAddrs;
+ }
+ } else if (holders == null)
+ newAddrs = parsedAddresses(clientCfg.getAddresses());
+
+ if (newAddrs == null) {
+ finishChannelsReInit = System.currentTimeMillis();
Review comment:
NL
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
+
+ } catch (Throwable ex) {
+ fut.completeExceptionally(ex);
+ return;
}
- if (err instanceof ClientConnectionException) {
- onChannelFailure(ch);
+ ClientChannel ch = chAndAttempts.getKey();
- if (failure == null)
- failure = (ClientConnectionException) err;
- else
- failure.addSuppressed(err);
+ ch
+ .serviceAsync(op, payloadWriter, payloadReader)
+ .handle((res, err) -> {
+ if (err == null) {
+ fut.complete(res);
Review comment:
NL
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
+
+ } catch (Throwable ex) {
+ fut.completeExceptionally(ex);
+ return;
}
- if (err instanceof ClientConnectionException) {
- onChannelFailure(ch);
+ ClientChannel ch = chAndAttempts.getKey();
- if (failure == null)
- failure = (ClientConnectionException) err;
- else
- failure.addSuppressed(err);
+ ch
+ .serviceAsync(op, payloadWriter, payloadReader)
+ .handle((res, err) -> {
+ if (err == null) {
+ fut.complete(res);
- if (chIdx == null)
- chIdx = new AtomicInteger();
+ return null;
+ }
- while (chIdx.incrementAndGet() < channels.length) {
- try {
- ch = channel();
+ ClientConnectionException failure0 = failure;
+
+ if (err instanceof ClientConnectionException) {
+ try {
+ // Will try to reinit channels if topology changed.
+ onChannelFailure(ch);
+ }
+ catch (Throwable ex) {
+ fut.completeExceptionally(ex);
Review comment:
NL
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
+
+ } catch (Throwable ex) {
+ fut.completeExceptionally(ex);
+ return;
}
- if (err instanceof ClientConnectionException) {
- onChannelFailure(ch);
+ ClientChannel ch = chAndAttempts.getKey();
- if (failure == null)
- failure = (ClientConnectionException) err;
- else
- failure.addSuppressed(err);
+ ch
+ .serviceAsync(op, payloadWriter, payloadReader)
+ .handle((res, err) -> {
+ if (err == null) {
+ fut.complete(res);
- if (chIdx == null)
- chIdx = new AtomicInteger();
+ return null;
+ }
- while (chIdx.incrementAndGet() < channels.length) {
- try {
- ch = channel();
+ ClientConnectionException failure0 = failure;
+
+ if (err instanceof ClientConnectionException) {
+ try {
+ // Will try to reinit channels if topology changed.
+ onChannelFailure(ch);
+ }
+ catch (Throwable ex) {
+ fut.completeExceptionally(ex);
+ return null;
+ }
- ClientConnectionException failure0 = failure;
- AtomicInteger chIdx0 = chIdx;
- ClientChannel ch0 = ch;
+ if (failure0 == null)
+ failure0 = (ClientConnectionException)err;
+ else
+ failure0.addSuppressed(err);
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res2, err2) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, failure0, chIdx0, ch0, res2, err2));
+ int leftAttempts = attemptsLimit - chAndAttempts.getValue();
+ // If it is a first retry then reset attempts (as for initialization we use only 1 attempt).
+ if (failure == null)
+ leftAttempts = getRetryLimit() - 1;
+
+ if (leftAttempts > 0) {
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, leftAttempts, failure0);
+ return null;
+ }
+ }
+ else {
+ fut.completeExceptionally(err instanceof ClientException ? err : new ClientException(err));
Review comment:
NL
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
+
+ } catch (Throwable ex) {
+ fut.completeExceptionally(ex);
Review comment:
What if `failure` already not null here? We will lose this information?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -606,16 +603,271 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /**
+ * Should the channel initialization be stopped.
+ */
+ private boolean shouldStopChannelsReinit() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ * @return {@code true} if holders are reinited and {@code false} if the initialization was interrupted.
+ */
+ synchronized boolean initChannelHolders(boolean force) {
+ List<ClientChannelHolder> holders = channels;
+
+ if (!force && holders != null)
+ return true;
+
+ startChannelsReInit = System.currentTimeMillis();
+
+ // Enable parallel threads to schedule new init of channel holders.
+ scheduledChannelsReinit.set(false);
+
+ Map<InetSocketAddress, Integer> newAddrs = null;
+
+ if (clientCfg.getAddressesFinder() != null) {
+ String[] hostAddrs = clientCfg.getAddressesFinder().getAddresses();
+
+ if (hostAddrs.length == 0)
+ throw new ClientException("Empty addresses");
+
+ if (!Arrays.equals(hostAddrs, prevHostAddrs)) {
+ newAddrs = parsedAddresses(hostAddrs);
+ prevHostAddrs = hostAddrs;
+ }
+ } else if (holders == null)
+ newAddrs = parsedAddresses(clientCfg.getAddresses());
+
+ if (newAddrs == null) {
+ finishChannelsReInit = System.currentTimeMillis();
+ return true;
+ }
+
+ Map<InetSocketAddress, ClientChannelHolder> curAddrs = new HashMap<>();
+ Set<InetSocketAddress> allAddrs = new HashSet<>(newAddrs.keySet());
+
+ if (holders != null) {
+ for (int i = 0; i < holders.size(); i++) {
+ ClientChannelHolder h = holders.get(i);
+
+ curAddrs.put(h.chCfg.getAddress(), h);
+ allAddrs.add(h.chCfg.getAddress());
+ }
+ }
+
+ List<ClientChannelHolder> reinitHolders = new ArrayList<>();
+
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currDfltHolder = null;
+
+ int idx = curChIdx;
+
+ if (idx != -1)
+ currDfltHolder = holders.get(idx);
+
+ for (InetSocketAddress addr : allAddrs) {
+ if (shouldStopChannelsReinit())
+ return false;
+
+ // Obsolete addr, to be removed.
+ if (!newAddrs.containsKey(addr)) {
+ curAddrs.get(addr).close();
+
+ continue;
+ }
+
+ // Create new holders for new addrs.
+ if (!curAddrs.containsKey(addr)) {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+
+ for (int i = 0; i < newAddrs.get(addr); i++)
+ reinitHolders.add(hld);
+
+ continue;
+ }
+
+ // This holder is up to date.
+ ClientChannelHolder hld = curAddrs.get(addr);
+
+ for (int i = 0; i < newAddrs.get(addr); i++)
+ reinitHolders.add(hld);
+
+ if (hld == currDfltHolder)
+ dfltChannelIdx = reinitHolders.size() - 1;
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(reinitHolders.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels = reinitHolders;
+ curChIdx = dfltChannelIdx;
+ }
+ finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+
+ finishChannelsReInit = System.currentTimeMillis();
Review comment:
NL
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
Review comment:
I think it's better to control attempts count in one method. I propose to rewrite this method without using `applyOnDefaultChannel` and rewrite `applyOnDefaultChannel` to return only result but not `T2` with attempts counter (help GC a little bit).
WDYT?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -606,16 +603,271 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /**
+ * Should the channel initialization be stopped.
+ */
+ private boolean shouldStopChannelsReinit() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ * @return {@code true} if holders are reinited and {@code false} if the initialization was interrupted.
+ */
+ synchronized boolean initChannelHolders(boolean force) {
+ List<ClientChannelHolder> holders = channels;
+
+ if (!force && holders != null)
+ return true;
+
+ startChannelsReInit = System.currentTimeMillis();
+
+ // Enable parallel threads to schedule new init of channel holders.
+ scheduledChannelsReinit.set(false);
+
+ Map<InetSocketAddress, Integer> newAddrs = null;
+
+ if (clientCfg.getAddressesFinder() != null) {
+ String[] hostAddrs = clientCfg.getAddressesFinder().getAddresses();
+
+ if (hostAddrs.length == 0)
+ throw new ClientException("Empty addresses");
+
+ if (!Arrays.equals(hostAddrs, prevHostAddrs)) {
+ newAddrs = parsedAddresses(hostAddrs);
+ prevHostAddrs = hostAddrs;
+ }
+ } else if (holders == null)
+ newAddrs = parsedAddresses(clientCfg.getAddresses());
+
+ if (newAddrs == null) {
+ finishChannelsReInit = System.currentTimeMillis();
+ return true;
+ }
+
+ Map<InetSocketAddress, ClientChannelHolder> curAddrs = new HashMap<>();
+ Set<InetSocketAddress> allAddrs = new HashSet<>(newAddrs.keySet());
+
+ if (holders != null) {
+ for (int i = 0; i < holders.size(); i++) {
+ ClientChannelHolder h = holders.get(i);
+
+ curAddrs.put(h.chCfg.getAddress(), h);
+ allAddrs.add(h.chCfg.getAddress());
+ }
+ }
+
+ List<ClientChannelHolder> reinitHolders = new ArrayList<>();
+
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currDfltHolder = null;
+
+ int idx = curChIdx;
+
+ if (idx != -1)
+ currDfltHolder = holders.get(idx);
+
+ for (InetSocketAddress addr : allAddrs) {
+ if (shouldStopChannelsReinit())
+ return false;
+
+ // Obsolete addr, to be removed.
+ if (!newAddrs.containsKey(addr)) {
+ curAddrs.get(addr).close();
+
+ continue;
+ }
+
+ // Create new holders for new addrs.
+ if (!curAddrs.containsKey(addr)) {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+
+ for (int i = 0; i < newAddrs.get(addr); i++)
+ reinitHolders.add(hld);
+
+ continue;
+ }
+
+ // This holder is up to date.
+ ClientChannelHolder hld = curAddrs.get(addr);
+
+ for (int i = 0; i < newAddrs.get(addr); i++)
+ reinitHolders.add(hld);
+
+ if (hld == currDfltHolder)
+ dfltChannelIdx = reinitHolders.size() - 1;
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(reinitHolders.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels = reinitHolders;
+ curChIdx = dfltChannelIdx;
+ }
+ finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+
+ finishChannelsReInit = System.currentTimeMillis();
+ return true;
+ }
+
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void channelsInit(boolean force) {
+ if (!force && channels != null)
+ return;
+
+ // Do not establish connections if interrupted.
+ if (!initChannelHolders(force))
+ return;
+
+ // Apply no-op function. Establish default channel connection.
+ applyOnDefaultChannel(channel -> null);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = hld != null ? hld.getOrCreateChannel() : null;
+
+ if (channel != null)
+ return function.apply(channel);
+
Review comment:
Redundant NL
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -606,16 +603,271 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /**
+ * Should the channel initialization be stopped.
+ */
+ private boolean shouldStopChannelsReinit() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ * @return {@code true} if holders are reinited and {@code false} if the initialization was interrupted.
+ */
+ synchronized boolean initChannelHolders(boolean force) {
+ List<ClientChannelHolder> holders = channels;
+
+ if (!force && holders != null)
+ return true;
+
+ startChannelsReInit = System.currentTimeMillis();
+
+ // Enable parallel threads to schedule new init of channel holders.
+ scheduledChannelsReinit.set(false);
+
+ Map<InetSocketAddress, Integer> newAddrs = null;
+
+ if (clientCfg.getAddressesFinder() != null) {
+ String[] hostAddrs = clientCfg.getAddressesFinder().getAddresses();
+
+ if (hostAddrs.length == 0)
+ throw new ClientException("Empty addresses");
+
+ if (!Arrays.equals(hostAddrs, prevHostAddrs)) {
+ newAddrs = parsedAddresses(hostAddrs);
+ prevHostAddrs = hostAddrs;
+ }
+ } else if (holders == null)
+ newAddrs = parsedAddresses(clientCfg.getAddresses());
+
+ if (newAddrs == null) {
+ finishChannelsReInit = System.currentTimeMillis();
+ return true;
+ }
+
+ Map<InetSocketAddress, ClientChannelHolder> curAddrs = new HashMap<>();
+ Set<InetSocketAddress> allAddrs = new HashSet<>(newAddrs.keySet());
+
+ if (holders != null) {
+ for (int i = 0; i < holders.size(); i++) {
+ ClientChannelHolder h = holders.get(i);
+
+ curAddrs.put(h.chCfg.getAddress(), h);
+ allAddrs.add(h.chCfg.getAddress());
+ }
+ }
+
+ List<ClientChannelHolder> reinitHolders = new ArrayList<>();
+
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currDfltHolder = null;
+
+ int idx = curChIdx;
+
+ if (idx != -1)
+ currDfltHolder = holders.get(idx);
+
+ for (InetSocketAddress addr : allAddrs) {
+ if (shouldStopChannelsReinit())
+ return false;
+
+ // Obsolete addr, to be removed.
+ if (!newAddrs.containsKey(addr)) {
+ curAddrs.get(addr).close();
+
+ continue;
+ }
+
+ // Create new holders for new addrs.
+ if (!curAddrs.containsKey(addr)) {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+
+ for (int i = 0; i < newAddrs.get(addr); i++)
+ reinitHolders.add(hld);
+
+ continue;
+ }
+
+ // This holder is up to date.
+ ClientChannelHolder hld = curAddrs.get(addr);
+
+ for (int i = 0; i < newAddrs.get(addr); i++)
+ reinitHolders.add(hld);
+
+ if (hld == currDfltHolder)
+ dfltChannelIdx = reinitHolders.size() - 1;
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(reinitHolders.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels = reinitHolders;
+ curChIdx = dfltChannelIdx;
+ }
+ finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+
+ finishChannelsReInit = System.currentTimeMillis();
+ return true;
+ }
+
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void channelsInit(boolean force) {
+ if (!force && channels != null)
Review comment:
Currently, flag `force` is useless (at least for production code, but perhaps for test code too). There are 3 calls of this method from production code, first - right after constructor (when `channels == null`), two other calls with `force == true`.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -512,80 +504,81 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
return ranges.stream()
.flatMap(r -> IntStream
.rangeClosed(r.portFrom(), r.portTo()).boxed()
- .map(p -> new InetSocketAddress(r.host(), p))
+ .map(p -> InetSocketAddress.createUnresolved(r.host(), p))
)
- .collect(Collectors.toList());
+ .collect(Collectors.toMap(a -> a, a -> 1, Integer::sum));
}
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
+ /**
+ * Roll current default channel if specified holder equals to it.
+ */
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
Review comment:
NL
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -512,80 +504,81 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
return ranges.stream()
.flatMap(r -> IntStream
.rangeClosed(r.portFrom(), r.portTo()).boxed()
- .map(p -> new InetSocketAddress(r.host(), p))
+ .map(p -> InetSocketAddress.createUnresolved(r.host(), p))
)
- .collect(Collectors.toList());
+ .collect(Collectors.toMap(a -> a, a -> 1, Integer::sum));
}
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
+ /**
+ * Roll current default channel if specified holder equals to it.
+ */
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
+ int idx = curChIdx;
+ List<ClientChannelHolder> holders = channels;
- throw e;
- }
- }
+ ClientChannelHolder dfltHld = holders.get(idx);
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ if (dfltHld == hld) {
+ idx += 1;
+
+ if (idx >= holders.size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On current channel failure.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
+ private void onChannelFailure(ClientChannel ch) {
// There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
// when current index was changed and no other wrong channel will be closed by current thread because
// onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ onChannelFailure(channels.get(curChIdx), ch);
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ chFailLsnrs.forEach(Runnable::run);
+
+ if (scheduledChannelsReinit.get())
+ channelsInit(true);
+ else
+ rollCurrentChannel(hld);
Review comment:
Why do we skip rollCurrentChannel if scheduledChannelsReinit? I think we should roll channel in any case.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -512,80 +504,81 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
return ranges.stream()
.flatMap(r -> IntStream
.rangeClosed(r.portFrom(), r.portTo()).boxed()
- .map(p -> new InetSocketAddress(r.host(), p))
+ .map(p -> InetSocketAddress.createUnresolved(r.host(), p))
)
- .collect(Collectors.toList());
+ .collect(Collectors.toMap(a -> a, a -> 1, Integer::sum));
}
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
+ /**
+ * Roll current default channel if specified holder equals to it.
+ */
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
+ int idx = curChIdx;
+ List<ClientChannelHolder> holders = channels;
- throw e;
- }
- }
+ ClientChannelHolder dfltHld = holders.get(idx);
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ if (dfltHld == hld) {
+ idx += 1;
+
+ if (idx >= holders.size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On current channel failure.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
+ private void onChannelFailure(ClientChannel ch) {
// There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
// when current index was changed and no other wrong channel will be closed by current thread because
// onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ onChannelFailure(channels.get(curChIdx), ch);
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ chFailLsnrs.forEach(Runnable::run);
+
+ if (scheduledChannelsReinit.get())
Review comment:
Perhaps `&& !partitionAwarenessEnabled` should be added here (to avoid double reinit for partition awareness, first time here and second time in async thread). WDYT?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
+
+ } catch (Throwable ex) {
+ fut.completeExceptionally(ex);
+ return;
}
- if (err instanceof ClientConnectionException) {
- onChannelFailure(ch);
+ ClientChannel ch = chAndAttempts.getKey();
- if (failure == null)
- failure = (ClientConnectionException) err;
- else
- failure.addSuppressed(err);
+ ch
+ .serviceAsync(op, payloadWriter, payloadReader)
+ .handle((res, err) -> {
+ if (err == null) {
+ fut.complete(res);
- if (chIdx == null)
- chIdx = new AtomicInteger();
+ return null;
+ }
- while (chIdx.incrementAndGet() < channels.length) {
- try {
- ch = channel();
+ ClientConnectionException failure0 = failure;
+
+ if (err instanceof ClientConnectionException) {
+ try {
+ // Will try to reinit channels if topology changed.
+ onChannelFailure(ch);
+ }
+ catch (Throwable ex) {
+ fut.completeExceptionally(ex);
+ return null;
+ }
- ClientConnectionException failure0 = failure;
- AtomicInteger chIdx0 = chIdx;
- ClientChannel ch0 = ch;
+ if (failure0 == null)
+ failure0 = (ClientConnectionException)err;
+ else
+ failure0.addSuppressed(err);
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res2, err2) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, failure0, chIdx0, ch0, res2, err2));
+ int leftAttempts = attemptsLimit - chAndAttempts.getValue();
+ // If it is a first retry then reset attempts (as for initialization we use only 1 attempt).
+ if (failure == null)
+ leftAttempts = getRetryLimit() - 1;
+
+ if (leftAttempts > 0) {
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, leftAttempts, failure0);
Review comment:
NL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r494266222
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
+ */
+ @Test
+ public void testClientDiscoveryNodesJoin() throws Exception {
+ for (int i = 0; i < MAX_CLUSTER_SIZE; ++i) {
+ startGrid(i);
+ awaitPartitionMapExchange();
+
+ int[] workChannels = IntStream.rangeClosed(0, i).toArray();
+
+ if (i == 0)
+ initClient(getClientConfigurationWithDiscovery(), workChannels);
+ else {
+ detectTopologyChange();
+ awaitChannelsInit(workChannels);
+ }
+
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Test that client use channels to all running nodes while nodes stop
+ */
+ @Test
+ public void testClientDiscoveryNodesLeave() throws Exception {
+ startGrids(MAX_CLUSTER_SIZE);
+ awaitPartitionMapExchange();
+
+ initClient(getClientConfigurationWithDiscovery(), 0, 1, 2, 3);
+ detectTopologyChange();
+
+ for (int i = MAX_CLUSTER_SIZE - 1; i != 0; i--) {
+ int[] workChannels = IntStream.range(0, i).toArray();
+
+ channels[i] = null;
+ stopGrid(i);
+ awaitPartitionMapExchange();
+ detectTopologyChange();
+
+ awaitChannelsInit(workChannels);
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Checks that each request goes to right node.
+ */
+ private void testPartitionAwareness(int... chIdxs) {
+ ClientCache<Object, Object> clientCache = client.cache(PART_CACHE_NAME);
+ IgniteInternalCache<Object, Object> igniteCache = grid(0).context().cache().cache(PART_CACHE_NAME);
+
+ Map<TestTcpClientChannel, Boolean> channelHits = Arrays.stream(chIdxs).boxed()
+ .collect(Collectors.toMap(idx -> channels[idx], idx -> false));
+
+ for (int i = 0; i < KEY_CNT; i++) {
+ TestTcpClientChannel opCh = affinityChannel(i, igniteCache);
+
+ clientCache.put(i, i);
+
+ if (i == 0)
+ assertOpOnChannel(dfltCh, ClientOperation.CACHE_PARTITIONS);
+
+ assertOpOnChannel(opCh, ClientOperation.CACHE_PUT);
+ assertTrue(channelHits.containsKey(opCh));
+
+ channelHits.compute(opCh, (c, old) -> true);
+ }
+
+ assertFalse(channelHits.containsValue(false));
+ }
+
+ /**
+ * Provide ClientConfiguration with addrResolver that find all alive nodes
+ */
+ private ClientConfiguration getClientConfigurationWithDiscovery() {
+ ClientAddressFinder addrFinder = () ->
+ IgnitionEx.allGrids().stream().map(node -> {
+ int port = (Integer) node.cluster().localNode().attributes().get(CLIENT_LISTENER_PORT);
+ return "127.0.0.1:" + port;
+ }).toArray(String[]::new);
+
+ return new ClientConfiguration()
+ .setAddressesFinder(addrFinder)
+ .setPartitionAwarenessEnabled(true);
+ }
+
+ /**
+ * Trigger client to detect topology change
+ */
+ private void detectTopologyChange() {
+ // Send non-affinity request to detect topology change.
+ initDefaultChannel();
+ client.getOrCreateCache(PART_CACHE_NAME);
Review comment:
Sorry, you're correct. Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r501248612
##########
File path: modules/indexing/pom.xml
##########
@@ -133,6 +133,14 @@
<version>${guava.version}</version>
<scope>test</scope>
</dependency>
+
+ <dependency>
Review comment:
To reproduce it you need run `ClientTestSuite` within indexing module classpath.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r488674967
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
Review comment:
FIxed. Skip check flag `partitionAwareness` and just checks topology version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r483148056
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +116,33 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /**
+ * Try use other limited number of channels to send a request if default channel is not responding.
Review comment:
fixed.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ClientChannelHolder.java
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.ignite.client.ClientAuthenticationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * Channels holder.
Review comment:
Outdated.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ClientChannelHolder.java
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.ignite.client.ClientAuthenticationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * Channels holder.
+ */
+class ClientChannelHolder {
+ /** Channel configuration. */
+ final ClientChannelConfiguration chCfg;
Review comment:
fixed.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ClientChannelHolder.java
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.ignite.client.ClientAuthenticationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * Channels holder.
+ */
+class ClientChannelHolder {
+ /** Channel configuration. */
+ final ClientChannelConfiguration chCfg;
+
+ /** Channel. */
+ private volatile ClientChannel ch;
+
+ /** Address that holder is bind to (chCfg.addr) is not in use now. So close the holder */
+ private volatile boolean close;
+
+ /** Timestamps of reconnect retries. */
+ private final long[] reconnectRetries;
+
+ /** Channel factory. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
+
+ /** Callback invokes when new channel create */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate;
+
+ /** Callback invokes when channel close */
+ private final Consumer<ClientChannel> onChannelClose;
+
+ /**
+ * @param chCfg Channel config.
+ */
+ ClientChannelHolder(ClientChannelConfiguration chCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate,
+ Consumer<ClientChannel> onChannelClose) {
+ this.chCfg = chCfg;
+ this.chFactory = chFactory;
+ this.onChannelCreate = onChannelCreate;
+ this.onChannelClose = onChannelClose;
+
+ reconnectRetries = chCfg.getReconnectThrottlingRetries() > 0 && chCfg.getReconnectThrottlingPeriod() > 0L ?
+ new long[chCfg.getReconnectThrottlingRetries()] : null;
+ }
+
+ /**
+ * @return Whether reconnect throttling should be applied.
+ */
+ boolean applyReconnectionThrottling() {
Review comment:
fixed.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +65,16 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
/** Index of the current channel. */
- private int curChIdx;
+ private volatile int curChIdx = -1;
- /** Partition awareness enabled. */
- private final boolean partitionAwarenessEnabled;
+ /** Is all channels should be initialized at one moment. */
Review comment:
Outdated.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
Review comment:
Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] isapego commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
isapego commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482838479
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ClientChannelHolder.java
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.ignite.client.ClientAuthenticationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * Channels holder.
Review comment:
Please, add a description of what is this class for and how should it be used.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ClientChannelHolder.java
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.ignite.client.ClientAuthenticationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * Channels holder.
+ */
+class ClientChannelHolder {
+ /** Channel configuration. */
+ final ClientChannelConfiguration chCfg;
+
+ /** Channel. */
+ private volatile ClientChannel ch;
+
+ /** Address that holder is bind to (chCfg.addr) is not in use now. So close the holder */
+ private volatile boolean close;
+
+ /** Timestamps of reconnect retries. */
+ private final long[] reconnectRetries;
+
+ /** Channel factory. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
+
+ /** Callback invokes when new channel create */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate;
+
+ /** Callback invokes when channel close */
+ private final Consumer<ClientChannel> onChannelClose;
+
+ /**
+ * @param chCfg Channel config.
+ */
+ ClientChannelHolder(ClientChannelConfiguration chCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate,
+ Consumer<ClientChannel> onChannelClose) {
+ this.chCfg = chCfg;
+ this.chFactory = chFactory;
+ this.onChannelCreate = onChannelCreate;
+ this.onChannelClose = onChannelClose;
+
+ reconnectRetries = chCfg.getReconnectThrottlingRetries() > 0 && chCfg.getReconnectThrottlingPeriod() > 0L ?
+ new long[chCfg.getReconnectThrottlingRetries()] : null;
+ }
+
+ /**
+ * @return Whether reconnect throttling should be applied.
+ */
+ boolean applyReconnectionThrottling() {
Review comment:
`private` here and for other methods where apply.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +116,33 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /**
+ * Try use other limited number of channels to send a request if default channel is not responding.
+ * 0 means try use all configured channels before fail.
+ */
+ private int channelsAttemptsLimit = 0;
Review comment:
+1 for `retryLimit`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r501126090
##########
File path: modules/core/src/main/java/org/apache/ignite/client/ClientAddressFinder.java
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.client;
+
+/**
+ * This interface provides a list of addresses of Ignite server nodes within a cluster. Thin client uses the list to
Review comment:
Fixed. Also change javadoc for `ClientConfiguration.setAddresses`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r489181956
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
Review comment:
Fixed.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -62,16 +68,19 @@
private final Function<ClientChannelConfiguration, ClientChannel> chFactory;
/** Client channel holders for each configured address. */
- private final ClientChannelHolder[] channels;
+ private final AtomicReference<List<ClientChannelHolder>> channels = new AtomicReference<>();
/** Index of the current channel. */
- private int curChIdx;
+ private volatile int curChIdx = -1;
/** Partition awareness enabled. */
private final boolean partitionAwarenessEnabled;
/** Cache partition awareness context. */
- private final ClientCacheAffinityContext affinityCtx;
+ private final ClientCacheAffinityContext affCtx;
Review comment:
Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r487743667
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
Review comment:
With StatefulSet it will be a static configuration, i.e. list of DNS names like "ignite-1, ignite-2, ignite-3". Also it's not prohibited to configure with DNS names in any other env (non-k8s).
`TcpClientChannel` creates socket with `cfg.getAddress().getHostName()` and it invokes reverse lookup even in case we specify address with hostnames.
Kubernetes have troubles with reverse lookup, as pods aren't aimed to be reverse resolved at all. There are some work in that direction, but the issue isn't resolved yet: https://github.com/coredns/coredns/issues/3888
So, I think we can't rely on this reverse lookup. So it could be case when DNS name isn't changed but the channel is broken and don't have changes to be fixed. So re-resolution is required.
I will check that resolved address isn't changed. And so skip next steps.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r485490144
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +116,30 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /** Retry limit. */
+ private int retryLimit = 0;
+
/**
* @return Host addresses.
*/
public String[] getAddresses() {
- return addrs;
+ return addrFinder.get();
Review comment:
`addrFinder` can be null
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/client/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.client;
+
+import java.net.InetAddress;
+import java.util.function.Supplier;
+import org.apache.ignite.internal.kubernetes.connection.KubernetesServiceAddressResolver;
+import org.apache.ignite.kubernetes.configuration.KubernetesConnectionConfiguration;
+
+/**
+ * Address finder for automatic lookup of Ignite server nodes running in Kubernetes environment. All Ignite nodes have
+ * to be deployed as Kubernetes pods in order to be found. Applications and Ignite nodes running outside of Kubernetes
+ * will not be able to reach the containerized counterparts.
+ * <p>
+ * The implementation is based on a distinct Kubernetes service. The name of the service must be set with
+ * {@code KubernetesConnectionConfiguration}. As for Ignite pods, it's recommended to label them in such a way that
+ * the service will target only server nodes.
+ * <p>
+ * The address finder, in its turn, will call this service to retrieve Ignite pods IP addresses. The port will be
+ * set later within {@link IgniteClient}. Make sure that all Ignite pods occupy a similar ClientConnector port,
+ * otherwise they will not be able to connect each other using this address finder.
+ * <p>
+ */
+public class ThinClientKubernetesAddressFinder implements Supplier<String[]> {
+ /** Kubernetes connection configuration */
+ private final KubernetesConnectionConfiguration cfg;
+
+ /** Constructor */
+ public ThinClientKubernetesAddressFinder(KubernetesConnectionConfiguration cfg) {
+ this.cfg = cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override public String[] get() {
+ return new KubernetesServiceAddressResolver(cfg)
Review comment:
Performance: do not create `KubernetesServiceAddressResolver` on every call, create it once in constructor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r489459220
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +135,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
-
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
affinityCtx = new ClientCacheAffinityContext(binary);
+ }
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
-
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
Review comment:
But it will be async. We should throw an exception if none of addresses are valid.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r483148654
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/kubernetes/KubernetesServiceAddressResolver.java
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.kubernetes;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.URL;
+import java.net.UnknownHostException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.security.SecureRandom;
+import java.security.cert.X509Certificate;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import javax.net.ssl.HostnameVerifier;
+import javax.net.ssl.HttpsURLConnection;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSession;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.internal.util.typedef.internal.U;
+
+/**
+ * The class is responsible to fetch list of IP address for all pods that runs the specified kubernetes service.
+ */
+public class KubernetesServiceAddressResolver {
Review comment:
This class used both by IpFinder and AddressFinder that are in different packages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r487073068
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
Review comment:
Perhaps code will be much more simple with regular loop over holders without streams and intermediate structures.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r501126189
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.testframework.GridTestUtils;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** */
+ private final String[] dfltAddrs = new String[]{"127.0.0.1:10800", "127.0.0.1:10801", "127.0.0.1:10802"};
+
+ /** Checks that it is possible configure addresses with duplication (for load balancing). */
Review comment:
Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r487017283
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +126,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
-
- affinityCtx = new ClientCacheAffinityContext(binary);
-
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
+ affCtx = new ClientCacheAffinityContext(binary);
+ }
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
+ applyOnDefaultChannel(channel -> {
+ // do nothing, just trigger channel connection.
Review comment:
Here and below, please use Ignite code style for comments (start with uppercase later, end with a point).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
Review comment:
Can we skip further steps if `clientCfg.getAddresses()` is not changed? (for example, in case of static addresses configuration you will always reinit holders, but it's redundant)
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -356,7 +318,7 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
/**
* @return host:port_range address lines parsed as {@link InetSocketAddress}.
*/
- private static List<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
+ private Set<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
Review comment:
Let's keep it `static`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
Review comment:
I'm not sure that replacing `List` to `Set` it's a good idea here (perhaps you should just copy list to set here locally). Using `List` you can solve some user problems which can't be solved with `Set`. For example, using `list` you can make some server nodes to be used more often than another. Also, by having a single duplicated address you can workaround some known issues.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
Review comment:
`new ArrayList<>()` -> `Collections.emptyList()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(
+ new ClientChannelConfiguration(clientCfg, addr), chFactory, onChannelCreate, onChannelClose);
+
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
- affinityCtx = new ClientCacheAffinityContext(binary);
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
- ClientConnectionException lastEx = null;
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
- for (int i = 0; i < channels.length; i++) {
+ curChannelsGuard.writeLock().lock();
Review comment:
Agree with Pavel, we don't need `AtomicReference`, it's too verbose and we don't use any atomic features (no CAS operations, only get and set). Can be replaced with volatile variable (List or array).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
Review comment:
`force` flag is only set when partition awareness is used, but I think sometimes we should reinit holders even without partition awareness. For example, if we use Kubernetes discovery and new servers were started, the client will use only addresses of server nodes that were available at the time of client start.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
Review comment:
Perhaps code will be much more simple with regular loop over holders without streams and intermediate structures.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +126,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
-
- affinityCtx = new ClientCacheAffinityContext(binary);
-
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
+ affCtx = new ClientCacheAffinityContext(binary);
+ }
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
+ applyOnDefaultChannel(channel -> {
+ // do nothing, just trigger channel connection.
Review comment:
Here and below, please use Ignite code style for comments (start with uppercase later, end with a point).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
Review comment:
Can we skip further steps if `clientCfg.getAddresses()` is not changed? (for example, in case of static addresses configuration you will always reinit holders, but it's redundant)
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -356,7 +318,7 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
/**
* @return host:port_range address lines parsed as {@link InetSocketAddress}.
*/
- private static List<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
+ private Set<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
Review comment:
Let's keep it `static`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
Review comment:
I'm not sure that replacing `List` to `Set` it's a good idea here (perhaps you should just copy list to set here locally). Using `List` you can solve some user problems which can't be solved with `Set`. For example, using `list` you can make some server nodes to be used more often than another. Also, by having a single duplicated address you can workaround some known issues.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
Review comment:
`new ArrayList<>()` -> `Collections.emptyList()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(
+ new ClientChannelConfiguration(clientCfg, addr), chFactory, onChannelCreate, onChannelClose);
+
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
- affinityCtx = new ClientCacheAffinityContext(binary);
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
- ClientConnectionException lastEx = null;
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
- for (int i = 0; i < channels.length; i++) {
+ curChannelsGuard.writeLock().lock();
Review comment:
Agree with Pavel, we don't need `AtomicReference`, it's too verbose and we don't use any atomic features (no CAS operations, only get and set). Can be replaced with volatile variable (List or array).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
Review comment:
`force` flag is only set when partition awareness is used, but I think sometimes we should reinit holders even without partition awareness. For example, if we use Kubernetes discovery and new servers were started, the client will use only addresses of server nodes that were available at the time of client start.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
Review comment:
Perhaps code will be much more simple with regular loop over holders without streams and intermediate structures.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +126,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
-
- affinityCtx = new ClientCacheAffinityContext(binary);
-
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
+ affCtx = new ClientCacheAffinityContext(binary);
+ }
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
+ applyOnDefaultChannel(channel -> {
+ // do nothing, just trigger channel connection.
Review comment:
Here and below, please use Ignite code style for comments (start with uppercase later, end with a point).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
Review comment:
Can we skip further steps if `clientCfg.getAddresses()` is not changed? (for example, in case of static addresses configuration you will always reinit holders, but it's redundant)
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -356,7 +318,7 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
/**
* @return host:port_range address lines parsed as {@link InetSocketAddress}.
*/
- private static List<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
+ private Set<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
Review comment:
Let's keep it `static`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
Review comment:
I'm not sure that replacing `List` to `Set` it's a good idea here (perhaps you should just copy list to set here locally). Using `List` you can solve some user problems which can't be solved with `Set`. For example, using `list` you can make some server nodes to be used more often than another. Also, by having a single duplicated address you can workaround some known issues.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
Review comment:
`new ArrayList<>()` -> `Collections.emptyList()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(
+ new ClientChannelConfiguration(clientCfg, addr), chFactory, onChannelCreate, onChannelClose);
+
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
- affinityCtx = new ClientCacheAffinityContext(binary);
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
- ClientConnectionException lastEx = null;
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
- for (int i = 0; i < channels.length; i++) {
+ curChannelsGuard.writeLock().lock();
Review comment:
Agree with Pavel, we don't need `AtomicReference`, it's too verbose and we don't use any atomic features (no CAS operations, only get and set). Can be replaced with volatile variable (List or array).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
Review comment:
`force` flag is only set when partition awareness is used, but I think sometimes we should reinit holders even without partition awareness. For example, if we use Kubernetes discovery and new servers were started, the client will use only addresses of server nodes that were available at the time of client start.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
Review comment:
Perhaps code will be much more simple with regular loop over holders without streams and intermediate structures.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +126,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
-
- affinityCtx = new ClientCacheAffinityContext(binary);
-
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
+ affCtx = new ClientCacheAffinityContext(binary);
+ }
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
+ applyOnDefaultChannel(channel -> {
+ // do nothing, just trigger channel connection.
Review comment:
Here and below, please use Ignite code style for comments (start with uppercase later, end with a point).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
Review comment:
Can we skip further steps if `clientCfg.getAddresses()` is not changed? (for example, in case of static addresses configuration you will always reinit holders, but it's redundant)
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -356,7 +318,7 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
/**
* @return host:port_range address lines parsed as {@link InetSocketAddress}.
*/
- private static List<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
+ private Set<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
Review comment:
Let's keep it `static`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
Review comment:
I'm not sure that replacing `List` to `Set` it's a good idea here (perhaps you should just copy list to set here locally). Using `List` you can solve some user problems which can't be solved with `Set`. For example, using `list` you can make some server nodes to be used more often than another. Also, by having a single duplicated address you can workaround some known issues.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
Review comment:
`new ArrayList<>()` -> `Collections.emptyList()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(
+ new ClientChannelConfiguration(clientCfg, addr), chFactory, onChannelCreate, onChannelClose);
+
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
- affinityCtx = new ClientCacheAffinityContext(binary);
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
- ClientConnectionException lastEx = null;
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
- for (int i = 0; i < channels.length; i++) {
+ curChannelsGuard.writeLock().lock();
Review comment:
Agree with Pavel, we don't need `AtomicReference`, it's too verbose and we don't use any atomic features (no CAS operations, only get and set). Can be replaced with volatile variable (List or array).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
Review comment:
`force` flag is only set when partition awareness is used, but I think sometimes we should reinit holders even without partition awareness. For example, if we use Kubernetes discovery and new servers were started, the client will use only addresses of server nodes that were available at the time of client start.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
Review comment:
Perhaps code will be much more simple with regular loop over holders without streams and intermediate structures.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r505255089
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
+
+ } catch (Throwable ex) {
+ fut.completeExceptionally(ex);
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482818928
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +116,33 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /**
+ * Try use other limited number of channels to send a request if default channel is not responding.
+ * 0 means try use all configured channels before fail.
+ */
+ private int channelsAttemptsLimit = 0;
Review comment:
I introduced this variable because number of configured nodes dynamically changes. Earlier user could just configure addresses with limited number of nodes. With k8s configuration number of nodes are changes. And in case of large number of non-responding nodes (cluster is down, for example) without this limit client will wait for a long time for an exception.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r488673640
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -381,78 +343,59 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
.rangeClosed(r.portFrom(), r.portTo()).boxed()
.map(p -> new InetSocketAddress(r.host(), p))
)
- .collect(Collectors.toList());
- }
-
- /** */
- private synchronized ClientChannel channel() {
- if (closed)
- throw new ClientException("Channel is closed");
-
- try {
- return channels[curChIdx].getOrCreateChannel();
- }
- catch (ClientConnectionException e) {
- rollCurrentChannel();
-
- throw e;
- }
- }
-
- /** */
- private synchronized void rollCurrentChannel() {
- if (++curChIdx >= channels.length)
- curChIdx = 0;
+ .collect(Collectors.toSet());
}
/**
- * On current channel failure.
+ * Roll current default channel if specified holder equals to it.
*/
- private synchronized void onChannelFailure(ClientChannel ch) {
- // There is nothing wrong if curChIdx was concurrently changed, since channel was closed by another thread
- // when current index was changed and no other wrong channel will be closed by current thread because
- // onChannelFailure checks channel binded to the holder before closing it.
- onChannelFailure(channels[curChIdx], ch);
-
- chFailLsnrs.forEach(Runnable::run);
+ private void rollCurrentChannel(ClientChannelHolder hld) {
+ curChannelsGuard.writeLock().lock();
+ try {
+ ClientChannelHolder dfltHld = channels.get().get(curChIdx);
+ if (dfltHld == hld) {
+ int idx = curChIdx + 1;
+ if (idx >= channels.get().size())
+ curChIdx = 0;
+ else
+ curChIdx = idx;
+ }
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
}
/**
* On channel of the specified holder failure.
*/
- private synchronized void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
- if (ch == hld.ch && ch != null) {
+ private void onChannelFailure(ClientChannelHolder hld, ClientChannel ch) {
+ if (hld != null && ch != null && ch == hld.ch)
hld.closeChannel();
- if (hld == channels[curChIdx])
- rollCurrentChannel();
- }
+ rollCurrentChannel(hld);
+
+ chFailLsnrs.forEach(Runnable::run);
}
/**
* Asynchronously try to establish a connection to all configured servers.
*/
- private void initAllChannelsAsync() {
- // Skip if there is already channels reinit scheduled.
- if (scheduledChannelsReinit.compareAndSet(false, true)) {
- asyncRunner.submit(
- () -> {
- scheduledChannelsReinit.set(false);
-
- for (ClientChannelHolder hld : channels) {
- if (scheduledChannelsReinit.get() || closed)
- return; // New reinit task scheduled or channel is closed.
-
- try {
- hld.getOrCreateChannel(true);
- }
- catch (Exception ignore) {
- // No-op.
- }
+ void initAllChannelsAsync() {
Review comment:
Fixed.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
Review comment:
FIxed.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
+ if (!force && channels.get() != null)
+ return;
+
+ // Skip if there is already channels reinit scheduled.
+ // Flag is set back when a thread comes in synchronized initChannelHolders
+ if (scheduledChannelsReinit.compareAndSet(false, true)) {
+ initChannelHolders(force);
+
+ if (partitionAwarenessEnabled)
+ initAllChannelsAsync();
+ }
+ }
+
+ /**
+ * Apply specified {@code function} on a channel corresponding to specified {@code nodeId}.
+ */
+ private <T> T applyOnNodeChannel(UUID nodeId, Function<ClientChannel, T> function) {
+ ClientChannelHolder hld = null;
+ ClientChannel channel = null;
+
+ try {
+ hld = nodeChannels.get(nodeId);
+
+ channel = Optional
Review comment:
Fixed.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
Review comment:
Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r482818928
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -115,18 +116,33 @@
/** Reconnect throttling retries. See {@code reconnectThrottlingPeriod}. */
private int reconnectThrottlingRetries = 3;
+ /**
+ * Try use other limited number of channels to send a request if default channel is not responding.
+ * 0 means try use all configured channels before fail.
+ */
+ private int channelsAttemptsLimit = 0;
Review comment:
I introduced this variable because number of configured nodes can dynamically changes. Earlier user could just configure addresses with limited number of nodes. With k8s configuration number of nodes are changes. And in case of large number of non-responding nodes (cluster is down, for example) without this limit client will wait for a long time for an exception.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504912894
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
+
+ } catch (Throwable ex) {
+ fut.completeExceptionally(ex);
+ return;
}
- if (err instanceof ClientConnectionException) {
- onChannelFailure(ch);
+ ClientChannel ch = chAndAttempts.getKey();
- if (failure == null)
- failure = (ClientConnectionException) err;
- else
- failure.addSuppressed(err);
+ ch
+ .serviceAsync(op, payloadWriter, payloadReader)
+ .handle((res, err) -> {
+ if (err == null) {
+ fut.complete(res);
Review comment:
Sorry, I don't see this case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r493536588
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ThinClientPartitionAwarenessDiscoveryTest.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientCache;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.IgnitionEx;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.junit.Test;
+
+import static org.apache.ignite.internal.processors.odbc.ClientListenerProcessor.CLIENT_LISTENER_PORT;
+
+/**
+ * Test partition awareness of thin client on changed topology.
+ */
+public class ThinClientPartitionAwarenessDiscoveryTest extends ThinClientAbstractPartitionAwarenessTest {
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+ }
+
+ /**
+ * Test that client use channels to all running nodes while new nodes start
+ */
+ @Test
+ public void testClientDiscoveryNodesJoin() throws Exception {
+ for (int i = 0; i < MAX_CLUSTER_SIZE; ++i) {
+ startGrid(i);
+ awaitPartitionMapExchange();
+
+ int[] workChannels = IntStream.rangeClosed(0, i).toArray();
+
+ if (i == 0)
+ initClient(getClientConfigurationWithDiscovery(), workChannels);
+ else {
+ detectTopologyChange();
+ awaitChannelsInit(workChannels);
+ }
+
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Test that client use channels to all running nodes while nodes stop
+ */
+ @Test
+ public void testClientDiscoveryNodesLeave() throws Exception {
+ startGrids(MAX_CLUSTER_SIZE);
+ awaitPartitionMapExchange();
+
+ initClient(getClientConfigurationWithDiscovery(), 0, 1, 2, 3);
+ detectTopologyChange();
+
+ for (int i = MAX_CLUSTER_SIZE - 1; i != 0; i--) {
+ int[] workChannels = IntStream.range(0, i).toArray();
+
+ channels[i] = null;
+ stopGrid(i);
+ awaitPartitionMapExchange();
+ detectTopologyChange();
+
+ awaitChannelsInit(workChannels);
+ testPartitionAwareness(workChannels);
+ }
+ }
+
+ /**
+ * Checks that each request goes to right node.
+ */
+ private void testPartitionAwareness(int... chIdxs) {
+ ClientCache<Object, Object> clientCache = client.cache(PART_CACHE_NAME);
+ IgniteInternalCache<Object, Object> igniteCache = grid(0).context().cache().cache(PART_CACHE_NAME);
+
+ Map<TestTcpClientChannel, Boolean> channelHits = Arrays.stream(chIdxs).boxed()
+ .collect(Collectors.toMap(idx -> channels[idx], idx -> false));
+
+ for (int i = 0; i < KEY_CNT; i++) {
+ TestTcpClientChannel opCh = affinityChannel(i, igniteCache);
+
+ clientCache.put(i, i);
+
+ if (i == 0)
+ assertOpOnChannel(dfltCh, ClientOperation.CACHE_PARTITIONS);
+
+ assertOpOnChannel(opCh, ClientOperation.CACHE_PUT);
+ assertTrue(channelHits.containsKey(opCh));
+
+ channelHits.compute(opCh, (c, old) -> true);
+ }
+
+ assertFalse(channelHits.containsValue(false));
+ }
+
+ /**
+ * Provide ClientConfiguration with addrResolver that find all alive nodes
+ */
+ private ClientConfiguration getClientConfigurationWithDiscovery() {
+ ClientAddressFinder addrFinder = () ->
+ IgnitionEx.allGrids().stream().map(node -> {
+ int port = (Integer) node.cluster().localNode().attributes().get(CLIENT_LISTENER_PORT);
+ return "127.0.0.1:" + port;
+ }).toArray(String[]::new);
+
+ return new ClientConfiguration()
+ .setAddressesFinder(addrFinder)
+ .setPartitionAwarenessEnabled(true);
+ }
+
+ /**
+ * Trigger client to detect topology change
+ */
+ private void detectTopologyChange() {
+ // Send non-affinity request to detect topology change.
+ initDefaultChannel();
+ client.getOrCreateCache(PART_CACHE_NAME);
Review comment:
It's not. I'm not sure why, but without additional request I don't detect topology change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r487017283
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -114,46 +126,32 @@
Function<ClientChannelConfiguration, ClientChannel> chFactory,
ClientConfiguration clientCfg,
IgniteBinary binary
- ) throws ClientException {
+ ) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
-
- channels = new ClientChannelHolder[addrs.size()];
-
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled();
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
-
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
-
- affinityCtx = new ClientCacheAffinityContext(binary);
-
- ClientConnectionException lastEx = null;
-
- for (int i = 0; i < channels.length; i++) {
- try {
- channels[curChIdx].getOrCreateChannel();
-
- if (partitionAwarenessEnabled)
- initAllChannelsAsync();
-
- return;
- } catch (ClientConnectionException e) {
- lastEx = e;
-
- rollCurrentChannel();
- }
- }
+ affCtx = new ClientCacheAffinityContext(binary);
+ }
- throw lastEx;
+ /**
+ * Establishing connections to servers. If partition awareness feature is enabled connections are created
+ * for every configured server. Otherwise only default channel is connected.
+ */
+ void initConnection() {
+ channelsInit(false);
+ if (!partitionAwarenessEnabled)
+ applyOnDefaultChannel(channel -> {
+ // do nothing, just trigger channel connection.
Review comment:
Here and below, please use Ignite code style for comments (start with uppercase later, end with a point).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
Review comment:
Can we skip further steps if `clientCfg.getAddresses()` is not changed? (for example, in case of static addresses configuration you will always reinit holders, but it's redundant)
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -356,7 +318,7 @@ private boolean affinityInfoIsUpToDate(int cacheId) {
/**
* @return host:port_range address lines parsed as {@link InetSocketAddress}.
*/
- private static List<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
+ private Set<InetSocketAddress> parseAddresses(String[] addrs) throws ClientException {
Review comment:
Let's keep it `static`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
Review comment:
I'm not sure that replacing `List` to `Set` it's a good idea here (perhaps you should just copy list to set here locally). Using `List` you can solve some user problems which can't be solved with `Set`. For example, using `list` you can make some server nodes to be used more often than another. Also, by having a single duplicated address you can workaround some known issues.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
Review comment:
`new ArrayList<>()` -> `Collections.emptyList()`
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -98,265 +101,291 @@
/** Channels reinit was scheduled. */
private final AtomicBoolean scheduledChannelsReinit = new AtomicBoolean();
- /** Affinity map update is in progress. */
- private final AtomicBoolean affinityUpdateInProgress = new AtomicBoolean();
-
/** Channel is closed. */
private volatile boolean closed;
/** Fail (disconnect) listeners. */
- private ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
+ private final ArrayList<Runnable> chFailLsnrs = new ArrayList<>();
- /**
- * Constructor.
- */
- ReliableChannel(
- Function<ClientChannelConfiguration, ClientChannel> chFactory,
- ClientConfiguration clientCfg,
- IgniteBinary binary
- ) throws ClientException {
+ /** Fail (disconnect) listeners. */
+ private final ArrayList<Predicate<ClientChannel>> topChangeFilters = new ArrayList<>();
+
+ /** Guard channels and curChIdx together. */
+ private final ReadWriteLock curChannelsGuard = new ReentrantReadWriteLock();
+
+ /** Constructor. */
+ ReliableChannel(ClientConfiguration clientCfg,
+ Function<ClientChannelConfiguration, ClientChannel> chFactory,
+ boolean initAllChannels) {
if (chFactory == null)
throw new NullPointerException("chFactory");
if (clientCfg == null)
throw new NullPointerException("clientCfg");
+ this.clientCfg = clientCfg;
this.chFactory = chFactory;
+ this.initAllChannels = initAllChannels;
+ }
+
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
- List<InetSocketAddress> addrs = parseAddresses(clientCfg.getAddresses());
+ /** Callback is invoked after new ClientChannel has created. */
+ private final BiConsumer<ClientChannelHolder, ClientChannel> onChannelCreate = (holder, ch) -> {
+ ch.addTopologyChangeListener(channel -> {
+ if (topChangeFilters.stream().allMatch(s -> s.test(channel)))
+ channelsInit(true);
+ });
- channels = new ClientChannelHolder[addrs.size()];
+ ch.addNotificationListener(this);
- for (int i = 0; i < channels.length; i++)
- channels[i] = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addrs.get(i)));
+ nodeChannels.values().remove(holder);
+ nodeChannels.put(ch.serverNodeId(), holder);
+ };
- curChIdx = new Random().nextInt(channels.length); // We already verified there is at least one address.
+ /** Callback is invoked after a ClientChannel has closed. */
+ private final Consumer<ClientChannel> onChannelClose = ch -> {
+ for (Consumer<ClientChannel> lsnr : channelCloseLsnrs)
+ lsnr.accept(ch);
+ };
- partitionAwarenessEnabled = clientCfg.isPartitionAwarenessEnabled() && channels.length > 1;
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ List<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(
+ new ClientChannelConfiguration(clientCfg, addr), chFactory, onChannelCreate, onChannelClose);
+
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
- affinityCtx = new ClientCacheAffinityContext(binary);
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
- ClientConnectionException lastEx = null;
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
- for (int i = 0; i < channels.length; i++) {
+ curChannelsGuard.writeLock().lock();
Review comment:
Agree with Pavel, we don't need `AtomicReference`, it's too verbose and we don't use any atomic features (no CAS operations, only get and set). Can be replaced with volatile variable (List or array).
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -473,6 +415,196 @@ public void addChannelFailListener(Runnable chFailLsnr) {
chFailLsnrs.add(chFailLsnr);
}
+ /** Should the channel initialization be stopped. */
+ private boolean stopInitCondition() {
+ return scheduledChannelsReinit.get() || closed;
+ }
+
+ /**
+ * Init channel holders to all nodes.
+ * @param force enable to replace existing channels with new holders.
+ */
+ private synchronized void initChannelHolders(boolean force) {
+ // enable parallel threads to schedule new init of channel holders
+ scheduledChannelsReinit.set(false);
+
+ if (!force && channels.get() != null)
+ return;
+
+ Set<InetSocketAddress> resolvedAddrs = parseAddresses(clientCfg.getAddresses());
+
+ List<ClientChannelHolder> holders = Optional.ofNullable(channels.get()).orElse(new ArrayList<>());
+
+ // addr -> (holder, delete)
+ Map<InetSocketAddress, T2<ClientChannelHolder, Boolean>> addrs = holders.stream()
+ .collect(Collectors.toMap(
+ c -> c.chCfg.getAddress(),
+ c -> new T2<>(c, null)
+ ));
+
+ // mark for delete addrs that aren't provided by clientConfig now
+ addrs.keySet()
+ .stream()
+ .filter(addr -> !resolvedAddrs.contains(addr))
+ .forEach(addr -> addrs.get(addr).setValue(true));
+
+ // create new holders for new addrs
+ resolvedAddrs.stream()
+ .filter(addr -> !addrs.containsKey(addr))
+ .forEach(addr -> {
+ ClientChannelHolder hld = new ClientChannelHolder(new ClientChannelConfiguration(clientCfg, addr));
+ addrs.put(addr, new T2<>(hld, false));
+ });
+
+ if (!stopInitCondition()) {
+ List<ClientChannelHolder> list = new ArrayList<>();
+ // The variable holds a new index of default channel after topology change.
+ // Suppose that reuse of the channel is better than open new connection.
+ int dfltChannelIdx = -1;
+
+ ClientChannelHolder currHolder = null;
+ if (curChIdx != -1)
+ currHolder = channels.get().get(curChIdx);
+
+ for (T2<ClientChannelHolder, Boolean> t : addrs.values()) {
+ ClientChannelHolder hld = t.get1();
+ Boolean markForDelete = t.get2();
+
+ if (markForDelete == null) {
+ // this channel is still in use
+ list.add(hld);
+ if (hld == currHolder)
+ dfltChannelIdx = list.size() - 1;
+
+ }
+ else if (markForDelete) {
+ // this holder should be deleted now
+ nodeChannels.values().remove(hld);
+ hld.close();
+ }
+ else {
+ // this channel is new
+ list.add(hld);
+ }
+ }
+
+ if (dfltChannelIdx == -1)
+ dfltChannelIdx = new Random().nextInt(list.size());
+
+ curChannelsGuard.writeLock().lock();
+ try {
+ channels.set(list);
+ curChIdx = dfltChannelIdx;
+ } finally {
+ curChannelsGuard.writeLock().unlock();
+ }
+ }
+ }
+
+ /** Initialization of channels. */
+ private void channelsInit(boolean force) {
Review comment:
`force` flag is only set when partition awareness is used, but I think sometimes we should reinit holders even without partition awareness. For example, if we use Kubernetes discovery and new servers were started, the client will use only addresses of server nodes that were available at the time of client start.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r487072864
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
##########
@@ -33,9 +36,16 @@
/** Serial version uid. */
private static final long serialVersionUID = 0L;
- /** @serial Server addresses. */
+ /**
+ * @serial Server addresses.
+ * @deprecated deprecated in favor of {@link #addrFinder}. Keep it due to Serializable compatibility.
+ */
+ @Deprecated
private String[] addrs = null;
+ /** Server addresses finder. */
+ private transient Supplier<String[]> addrFinder;
Review comment:
I would prefer a dedicated interface - `ThinClientAddressFinder` or something like this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#issuecomment-702041555
@ptupitsyn hi! I've merge your changes related to async thin client API. I leverage on introduced by me `apply*` methods and then remove AtomicInteger you're used for the counter, also I've fix exception issue (when final exception missed if it's different from ClientConnectionException after retry) and add test for that.
Could you please review my changes again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r506132945
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -798,8 +801,10 @@ void channelsInit(boolean force) {
c = hld.getOrCreateChannel();
- if (c != null)
- return new T2<>(function.apply(c), attempt + 1);
+ if (c != null) {
+ attemptsCallback.accept(attempt + 1);
Review comment:
NL
##########
File path: modules/kubernetes/src/test/java/org/apache/ignite/internal/kubernetes/connection/KubernetesServiceAddressResolverTest.java
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.kubernetes.connection;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.kubernetes.configuration.KubernetesConnectionConfiguration;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.mockserver.integration.ClientAndServer;
+import org.mockserver.matchers.Times;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockserver.integration.ClientAndServer.startClientAndServer;
+import static org.mockserver.model.HttpRequest.request;
+import static org.mockserver.model.HttpResponse.response;
+
+/** Checks that correctly parse kubernetes json response. */
+public class KubernetesServiceAddressResolverTest {
+ /** Mock of kubernetes API. */
+ private static ClientAndServer mockServer;
+
+ /** */
+ private static final String namespace = "ns01";
+
+ /** */
+ private static final String service = "ignite";
+
+ /** */
+ @BeforeClass
+ public static void startServer() {
+ mockServer = startClientAndServer();
+ }
+
+ /** */
+ @AfterClass
+ public static void stopServer() {
+ mockServer.stop();
+ }
+
+ /** */
+ @Test
+ public void testCorrectParseKubernetesResponse() throws IOException {
+ // given
+ KubernetesServiceAddressResolver rslvr = prepareResolver(false);
+
+ mockSuccessServerResponse();
+
+ // when
+ Collection<InetAddress> result = rslvr.getServiceAddresses();
+
+ // then
+ List<String> ips = result.stream()
+ .map(InetAddress::getHostAddress)
+ .collect(Collectors.toList());
+
+ assertEquals(
+ Arrays.asList("10.1.1.1", "10.1.1.2", "10.1.1.4", "10.1.1.5", "10.1.1.7"),
+ ips
+ );
+ }
+
+ /** */
+ @Test
+ public void testCorrectParseKubernetesResponseWithIncludingNotReadyAddresses() throws IOException {
+ // given
+ KubernetesServiceAddressResolver rslvr = prepareResolver(true);
+
+ mockSuccessServerResponse();
+
+ // when
+ Collection<InetAddress> result = rslvr.getServiceAddresses();
+
+ // then
+ List<String> ips = result.stream()
+ .map(InetAddress::getHostAddress)
+ .collect(Collectors.toList());
+
+ assertEquals(
+ Arrays.asList("10.1.1.1", "10.1.1.2", "10.1.1.3", "10.1.1.4", "10.1.1.5", "10.1.1.6", "10.1.1.7"),
+ ips
+ );
+ }
+
+ /** */
+ @Test(expected = IgniteException.class)
+ public void testConnectionFailure() throws IOException {
+ // given
+ KubernetesServiceAddressResolver rslvr = prepareResolver(true);
+
+ mockFailureServerResponse();
+
+ rslvr.getServiceAddresses();
+ }
+
+ /** */
+ private KubernetesServiceAddressResolver prepareResolver(boolean includeNotReadyAddresses)
+ throws IOException
+ {
+ File account = File.createTempFile("kubernetes-test-account", "");
+ new FileWriter(account).write("account-token");
+ String accountFile = account.getAbsolutePath();
+
+ KubernetesConnectionConfiguration cfg = new KubernetesConnectionConfiguration();
+ cfg.setNamespace(namespace);
+ cfg.setServiceName(service);
+ cfg.setMasterUrl("https://localhost:" + mockServer.getLocalPort());
+ cfg.setAccountToken(accountFile);
+ cfg.setIncludeNotReadyAddresses(includeNotReadyAddresses);
+
+ return new KubernetesServiceAddressResolver(cfg);
+ }
+
+ /** */
+ private void mockFailureServerResponse() {
+ mockServer
+ .when(
+ request()
+ .withMethod("GET")
+ .withPath(String.format("/api/v1/namespaces/%s/endpoints/%s", namespace, service)),
+ Times.exactly(1)
+ )
+ .respond(
+ response()
+ .withStatusCode(401));
+ }
+
+ /** */
+ private void mockSuccessServerResponse() {
+ mockServer
+ .when(
+ request()
+ .withMethod("GET")
+ .withPath(String.format("/api/v1/namespaces/%s/endpoints/%s", namespace, service)),
+ Times.exactly(1)
+ )
+ .respond(
+ response()
+ .withStatusCode(200)
+ .withBody("{" +
+ " \"subsets\": [" +
+ " {" +
+ " \"addresses\": [" +
+ " {" +
+ " \"ip\": \"10.1.1.1\"" +
+ " }," +
+ " {" +
+ " \"ip\": \"10.1.1.2\"" +
+ " }" +
+ " ]," +
+ " \"notReadyAddresses\": [" +
+ " {" +
+ " \"ip\": \"10.1.1.3\"" +
+ " }" +
+ " ]" +
+ " }," +
+ " {" +
+ " \"addresses\": [" +
+ " {" +
+ " \"ip\": \"10.1.1.4\"" +
+ " }," +
+ " {" +
+ " \"ip\": \"10.1.1.5\"" +
+ " }" +
+ " ]," +
+ " \"notReadyAddresses\": [" +
+ " {" +
+ " \"ip\": \"10.1.1.6\"" +
+ " }" +
+ " ]" +
+ " }," +
+ " {" +
+ " \"addresses\": [" +
+ " {" +
+ " \"ip\": \"10.1.1.7\"" +
+ " }" +
+ " ]" +
+ " }" +
+ " ]" +
+ "}"
+ ));
+ }
+
+//
Review comment:
Clean it up, please
##########
File path: modules/kubernetes/src/test/java/org/apache/ignite/internal/kubernetes/connection/KubernetesServiceAddressResolverTest.java
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.kubernetes.connection;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.kubernetes.configuration.KubernetesConnectionConfiguration;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.mockserver.integration.ClientAndServer;
+import org.mockserver.matchers.Times;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockserver.integration.ClientAndServer.startClientAndServer;
+import static org.mockserver.model.HttpRequest.request;
+import static org.mockserver.model.HttpResponse.response;
+
+/** Checks that correctly parse kubernetes json response. */
+public class KubernetesServiceAddressResolverTest {
+ /** Mock of kubernetes API. */
+ private static ClientAndServer mockServer;
+
+ /** */
+ private static final String namespace = "ns01";
+
+ /** */
+ private static final String service = "ignite";
+
+ /** */
+ @BeforeClass
+ public static void startServer() {
+ mockServer = startClientAndServer();
+ }
+
+ /** */
+ @AfterClass
+ public static void stopServer() {
+ mockServer.stop();
+ }
+
+ /** */
+ @Test
+ public void testCorrectParseKubernetesResponse() throws IOException {
+ // given
+ KubernetesServiceAddressResolver rslvr = prepareResolver(false);
+
+ mockSuccessServerResponse();
+
+ // when
+ Collection<InetAddress> result = rslvr.getServiceAddresses();
+
+ // then
+ List<String> ips = result.stream()
+ .map(InetAddress::getHostAddress)
+ .collect(Collectors.toList());
+
+ assertEquals(
+ Arrays.asList("10.1.1.1", "10.1.1.2", "10.1.1.4", "10.1.1.5", "10.1.1.7"),
+ ips
+ );
+ }
+
+ /** */
+ @Test
+ public void testCorrectParseKubernetesResponseWithIncludingNotReadyAddresses() throws IOException {
+ // given
+ KubernetesServiceAddressResolver rslvr = prepareResolver(true);
+
+ mockSuccessServerResponse();
+
+ // when
+ Collection<InetAddress> result = rslvr.getServiceAddresses();
+
+ // then
+ List<String> ips = result.stream()
+ .map(InetAddress::getHostAddress)
+ .collect(Collectors.toList());
+
+ assertEquals(
+ Arrays.asList("10.1.1.1", "10.1.1.2", "10.1.1.3", "10.1.1.4", "10.1.1.5", "10.1.1.6", "10.1.1.7"),
+ ips
+ );
+ }
+
+ /** */
+ @Test(expected = IgniteException.class)
+ public void testConnectionFailure() throws IOException {
+ // given
+ KubernetesServiceAddressResolver rslvr = prepareResolver(true);
+
+ mockFailureServerResponse();
+
+ rslvr.getServiceAddresses();
+ }
+
+ /** */
+ private KubernetesServiceAddressResolver prepareResolver(boolean includeNotReadyAddresses)
+ throws IOException
+ {
+ File account = File.createTempFile("kubernetes-test-account", "");
+ new FileWriter(account).write("account-token");
Review comment:
Close FileWriter?
##########
File path: modules/kubernetes/src/test/java/org/apache/ignite/internal/kubernetes/connection/KubernetesServiceAddressResolverTest.java
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.kubernetes.connection;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.kubernetes.configuration.KubernetesConnectionConfiguration;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.mockserver.integration.ClientAndServer;
+import org.mockserver.matchers.Times;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockserver.integration.ClientAndServer.startClientAndServer;
+import static org.mockserver.model.HttpRequest.request;
+import static org.mockserver.model.HttpResponse.response;
+
+/** Checks that correctly parse kubernetes json response. */
+public class KubernetesServiceAddressResolverTest {
Review comment:
Why it is not included to the suite?
##########
File path: modules/kubernetes/src/test/java/org/apache/ignite/client/TestClusterClientConnection.java
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.client;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.stream.Collectors;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.kubernetes.configuration.KubernetesConnectionConfiguration;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.mockserver.integration.ClientAndServer;
+import org.mockserver.matchers.Times;
+
+import static org.mockserver.integration.ClientAndServer.startClientAndServer;
+import static org.mockserver.model.HttpRequest.request;
+import static org.mockserver.model.HttpResponse.response;
+
+/** Test that thin client connects to cluster with {@link ThinClientKubernetesAddressFinder}. */
+public class TestClusterClientConnection extends GridCommonAbstractTest {
+ /** Mock of kubernetes API. */
+ private static ClientAndServer mockServer;
+
+ /** */
+ private static final String namespace = "ns01";
+
+ /** */
+ private static final String service = "ignite";
+
+ /** */
+ @BeforeClass
+ public static void startServer() {
+ mockServer = startClientAndServer();
+ }
+
+ /** */
+ @AfterClass
+ public static void stopServer() {
+ mockServer.stop();
+ }
+
+ /** */
+ @After
+ public void tearDown() {
+ stopAllGrids();
+ }
+
+ /** */
+ @Test
+ public void testClientConnectsToCluster() throws Exception {
+ mockServerResponse();
+
+ IgniteEx crd = startGrid(getConfiguration());
+ String crdAddr = crd.localNode().addresses().iterator().next();
+
+ mockServerResponse(crdAddr);
+
+ ClientConfiguration ccfg = new ClientConfiguration();
+ ccfg.setAddressesFinder(new ThinClientKubernetesAddressFinder(prepareConfiguration()));
+
+ IgniteClient client = Ignition.startClient(ccfg);
+
+ ClientCache cache = client.createCache("cache");
+ cache.put(1, 2);
+ assertEquals(2, cache.get(1));
+ }
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration() throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration();
+
+ KubernetesConnectionConfiguration kccfg = prepareConfiguration();
+ TcpDiscoveryKubernetesIpFinder ipFinder = new TcpDiscoveryKubernetesIpFinder(kccfg);
+
+ TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
+ discoverySpi.setIpFinder(ipFinder);
+
+ cfg.setDiscoverySpi(discoverySpi);
+
+ cfg.setIgniteInstanceName(getTestIgniteInstanceName());
+
+ return cfg;
+ }
+
+ /** */
+ private void mockServerResponse(String... addrs) {
+ String ipAddrs = Arrays.stream(addrs)
+ .map(addr -> String.format("{\"ip\":\"%s\"}", addr))
+ .collect(Collectors.joining(","));
+
+ mockServer
+ .when(
+ request()
+ .withMethod("GET")
+ .withPath(String.format("/api/v1/namespaces/%s/endpoints/%s", namespace, service)),
+ Times.exactly(1)
+ )
+ .respond(
+ response()
+ .withStatusCode(200)
+ .withBody("{" +
+ " \"subsets\": [" +
+ " {" +
+ " \"addresses\": [" +
+ " " + ipAddrs +
+ " ]" +
+ " }" +
+ " ]" +
+ "}"
+ ));
+ }
+
+ /** */
+ private KubernetesConnectionConfiguration prepareConfiguration()
+ throws IOException
+ {
+ File account = File.createTempFile("kubernetes-test-account", "");
+ new FileWriter(account).write("account-token");
Review comment:
Close FileWriter?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -218,18 +220,24 @@
Function<PayloadInputChannel, T> payloadReader,
int attemptsLimit,
ClientConnectionException failure) {
- T2<ClientChannel, Integer> chAndAttempts;
+ ClientChannel ch;
+ // Workaround to store used attempts value within lambda body.
+ int attemptsCnt[] = new int[1];
try {
- chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
-
+ ch = applyOnDefaultChannel(channel -> channel, attemptsLimit, v -> attemptsCnt[0] = v );
} catch (Throwable ex) {
+ if (failure != null) {
+ failure.addSuppressed(ex);
+ fut.completeExceptionally(failure);
Review comment:
NL
##########
File path: modules/kubernetes/pom.xml
##########
@@ -101,6 +101,25 @@
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.mock-server</groupId>
+ <artifactId>mockserver-netty</artifactId>
+ <version>5.11.1</version>
Review comment:
`<scope>test</scope>`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r483148493
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannelFacade.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.ignite.IgniteBinary;
+import org.apache.ignite.client.ClientAuthenticationException;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+
+/**
+ * Communication channel with failover and partition awareness.
+ */
+final class ReliableChannelFacade implements AutoCloseable {
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r493495522
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -461,9 +418,8 @@ private void initAllChannelsAsync() {
* @param ch Channel.
*/
private void onTopologyChanged(ClientChannel ch) {
- if (partitionAwarenessEnabled && affinityCtx.updateLastTopologyVersion(ch.serverTopologyVersion(),
- ch.serverNodeId()))
- initAllChannelsAsync();
+ if (affinityCtx.updateLastTopologyVersion(ch.serverTopologyVersion(), ch.serverNodeId()))
+ channelsInit(true);
Review comment:
> we can do it sync in sending thread on channel failure and if we detect topology change before
It can be wrong. Topology change can be triggered by joining node, so channel failure may never occur. I'll make async initialization in any case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504833394
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
Review comment:
You already get rid of AtomicInteger, and it's not required here anymore. Yes, there will be a little code duplication for iterating through channels, I think it's ok since it will make code more readable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r504774568
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/client/thin/ReliableChannel.java
##########
@@ -223,67 +203,78 @@
) throws ClientException, ClientError {
CompletableFuture<T> fut = new CompletableFuture<>();
- ClientChannel ch = channel();
-
- ch.serviceAsync(op, payloadWriter, payloadReader).handle((res, err) ->
- handleServiceAsync(op, payloadWriter, payloadReader, fut, null, null, ch, res, err));
+ // Use the only one attempt to avoid blocking async method.
+ handleServiceAsync(fut, op, payloadWriter, payloadReader, 1, null);
return new IgniteClientFutureImpl<>(fut);
}
/**
* Handles serviceAsync results and retries as needed.
*/
- private <T> Object handleServiceAsync(ClientOperation op,
- Consumer<PayloadOutputChannel> payloadWriter,
- Function<PayloadInputChannel, T> payloadReader,
- CompletableFuture<T> fut,
- ClientConnectionException failure,
- AtomicInteger chIdx,
- ClientChannel ch,
- T res,
- Throwable err) {
- if (err == null) {
- fut.complete(res);
- return null;
+ private <T> void handleServiceAsync(final CompletableFuture<T> fut,
+ ClientOperation op,
+ Consumer<PayloadOutputChannel> payloadWriter,
+ Function<PayloadInputChannel, T> payloadReader,
+ int attemptsLimit,
+ ClientConnectionException failure) {
+ T2<ClientChannel, Integer> chAndAttempts;
+
+ try {
+ chAndAttempts = applyOnDefaultChannel(channel -> channel, attemptsLimit);
Review comment:
Currently master branch do work approach you describe, and it leads to:
1. introducing additional AtomicInteger variable for attempts counter. Counter is required as it used in separate threads execute `handleServiceAsync` method;
2. code duplication for iterating through channels (for sync and async methods).
This is reasons to use T2. It's not pretty, but alternative isn't looks better, I think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r500178504
##########
File path: modules/core/src/main/java/org/apache/ignite/client/ClientAddressFinder.java
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.client;
+
+/**
+ * This interface provides a list of addresses of Ignite server nodes within a cluster. Thin client uses the list to
Review comment:
Please provide details:
* When and how often Ignite will call the implementation
* What are the requirements:
* Duplicates allowed or not
* Allowed formats (with port, without port, ipv4/ipv6?)
##########
File path: modules/core/src/main/java/org/apache/ignite/client/ClientAddressFinder.java
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.client;
+
+/**
+ * This interface provides a list of addresses of Ignite server nodes within a cluster. Thin client uses the list to
+ * route user requests. There are cases when the list is not static, for example in cloud environment. In such cases
+ * addresses of nodes and/or number of server nodes can change. Implementation of this interface should handle these.
+ *
+ * {@link org.apache.ignite.configuration.ClientConfiguration#setAddressesFinder(ClientAddressFinder)}
+ */
+public interface ClientAddressFinder {
+ /** @return Addresses of Ignite server nodes within a cluster. */
+ public String[] getServerAddresses();
Review comment:
Let's rename to `getAddresses` for consistency with `ClientConfiguration`.
##########
File path: modules/indexing/pom.xml
##########
@@ -133,6 +133,14 @@
<version>${guava.version}</version>
<scope>test</scope>
</dependency>
+
+ <dependency>
Review comment:
Seems to be unnecessary.
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.testframework.GridTestUtils;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** */
+ private final String[] dfltAddrs = new String[]{"127.0.0.1:10800", "127.0.0.1:10801", "127.0.0.1:10802"};
+
+ /** Checks that it is possible configure addresses with duplication (for load balancing). */
Review comment:
Here and below please use multiline javadoc for methods as we do everywhere else.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r501244181
##########
File path: modules/core/src/main/java/org/apache/ignite/client/ClientAddressFinder.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.client;
+
+/**
+ * This interface provides a list of addresses of Ignite server nodes within a cluster. Thin client uses the list to
+ * route user requests. There are cases when the list is not static, for example in cloud environment. In such cases
+ * addresses of nodes and/or number of server nodes can change. Implementation of this interface should handle these.
+ *
+ * Ignite waites for a topology change to trigger {@link #getAddresses()}. There are two modes of how soon Ignite calls
+ * it, lazy and eager, depending on whether the partition awareness feature is enabled. If the feature is enabled then
+ * Ignite calls the method for every topology change. Otherwise Ignite uses previous addresses until a first failure.
+ *
+ * {@link org.apache.ignite.configuration.ClientConfiguration#setPartitionAwarenessEnabled(boolean)}
+ * {@link org.apache.ignite.configuration.ClientConfiguration#setAddressesFinder(ClientAddressFinder)}
+ */
+public interface ClientAddressFinder {
+ /**
+ * Get addresses of Ignite server nodes within a cluster. An address can be IPv4 address or hostname, with or
Review comment:
`ReliableChannel#parsedAddresses -> HostAndPortRange#parse` split address by colon and consider left part as hostname and right part as port.
I don't know is there other reasons do not use IPv6, need test for that. I suppose it should be a separate activity.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r501217118
##########
File path: modules/core/src/main/java/org/apache/ignite/client/ClientAddressFinder.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.client;
+
+/**
+ * This interface provides a list of addresses of Ignite server nodes within a cluster. Thin client uses the list to
+ * route user requests. There are cases when the list is not static, for example in cloud environment. In such cases
+ * addresses of nodes and/or number of server nodes can change. Implementation of this interface should handle these.
+ *
+ * Ignite waites for a topology change to trigger {@link #getAddresses()}. There are two modes of how soon Ignite calls
+ * it, lazy and eager, depending on whether the partition awareness feature is enabled. If the feature is enabled then
+ * Ignite calls the method for every topology change. Otherwise Ignite uses previous addresses until a first failure.
+ *
+ * {@link org.apache.ignite.configuration.ClientConfiguration#setPartitionAwarenessEnabled(boolean)}
+ * {@link org.apache.ignite.configuration.ClientConfiguration#setAddressesFinder(ClientAddressFinder)}
+ */
+public interface ClientAddressFinder {
+ /**
+ * Get addresses of Ignite server nodes within a cluster. An address can be IPv4 address or hostname, with or
Review comment:
I thought we support IPv6? Is there any reason not to?
##########
File path: modules/indexing/pom.xml
##########
@@ -133,6 +133,14 @@
<version>${guava.version}</version>
<scope>test</scope>
</dependency>
+
+ <dependency>
Review comment:
Are you sure? It compiles and runs on my machine without this change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r497490324
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
Review comment:
fixed.
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** */
+ private void checkDoesNotReinit(ClientConfiguration ccfg) {
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ List<ReliableChannel.ClientChannelHolder> originalChannels = rc.getChannelHolders();
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+
+ assertSame(originalChannels, newChannels);
+ IntStream.range(0, 3).forEach(i -> {
+ assertSame(originalChannels.get(i), newChannels.get(i));
+ assertFalse(originalChannels.get(i).isClosed());
+ });
+ assertEquals(3, newChannels.size());
+ }
+
+ /** Checks that node channels are persisted if channels are reinit with static address configuration. */
+ @Test
+ public void testNodeChannelsAreNotCleaned() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ // Trigger TestClientChannel creation.
+ rc.service(null, null, null);
+
+ assertEquals(1, rc.nodeChannels.size());
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+
+ assertEquals(1, rc.nodeChannels.size());
+ }
+
+ /** Checks that channels are changed (add new, remove old) and close channels if reinitialization performed. */
+ @Test
+ public void testDynamicAddressReinitializedCorrectly() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8003"));
+
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+
+ List<ReliableChannel.ClientChannelHolder> originChannels = Collections.unmodifiableList(rc.getChannelHolders());
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+
+ List<ReliableChannel.ClientChannelHolder> closedChannels = originChannels.stream()
+ .filter(r -> r.isClosed())
+ .collect(Collectors.toList());
+
+ assertEquals(2, closedChannels.size());
Review comment:
fixed.
##########
File path: modules/core/src/test/java/org/apache/ignite/internal/client/thin/ReliableChannelTest.java
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.ignite.client.ClientAddressFinder;
+import org.apache.ignite.client.ClientAuthorizationException;
+import org.apache.ignite.client.ClientConnectionException;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Class test ReliableChannel channels re-initialization.
+ */
+public class ReliableChannelTest {
+ /** Mock factory for creating new channels. */
+ private final Function<ClientChannelConfiguration, ClientChannel> chFactory = cfg -> new TestClientChannel();
+
+ /** Checks that channel holders are not reinited for static address configuration. */
+ @Test
+ public void testChannelsNotReinitForStaticAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002");
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** Checks that channel holders are not reinited if address finder return the same list of addresses. */
+ @Test
+ public void testChannelsNotReinitForStableDynamicAddressConfiguration() {
+ ClientConfiguration ccfg = new ClientConfiguration()
+ .setAddressesFinder(new TestAddressFinder("127.0.0.1:8000", "127.0.0.1:8001", "127.0.0.1:8002"));
+
+ checkDoesNotReinit(ccfg);
+ }
+
+ /** */
+ private void checkDoesNotReinit(ClientConfiguration ccfg) {
+ ReliableChannel rc = new ReliableChannel(chFactory, ccfg, null);
+ rc.initConnection();
+ List<ReliableChannel.ClientChannelHolder> originalChannels = rc.getChannelHolders();
+
+ // Imitate topology change.
+ rc.initChannelHolders(true);
+ List<ReliableChannel.ClientChannelHolder> newChannels = rc.getChannelHolders();
+
+ assertSame(originalChannels, newChannels);
+ IntStream.range(0, 3).forEach(i -> {
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8206: IGNITE-13204 Add Java thin client Kubernetes auto discovery
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8206:
URL: https://github.com/apache/ignite/pull/8206#discussion_r483148880
##########
File path: modules/kubernetes/src/main/java/org/apache/ignite/spi/discovery/tcp/ipfinder/kubernetes/ThinClientKubernetesAddressFinder.java
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes;
+
+import java.net.InetAddress;
+import java.util.function.Supplier;
+import org.apache.ignite.kubernetes.KubernetesConnectorConfigurator;
+import org.apache.ignite.kubernetes.KubernetesConnectorDefaults;
+import org.apache.ignite.kubernetes.KubernetesServiceAddressResolver;
+
+/**
+ * Address finder for automatic lookup of Ignite nodes running in Kubernetes environment. All Ignite nodes have to
+ * deployed as Kubernetes pods in order to be found. Applications and Ignite nodes running outside of Kubernetes
+ * will not be able to reach the containerized counterparts.
+ * <p>
+ * The implementation is based on a distinct Kubernetes service that has to be created and should be deployed prior
+ * Ignite nodes startup. The service will maintain a list of all endpoints (internal IP addresses) of all containerized
+ * Ignite pods running so far. The name of the service must be equal to {@link #setServiceName(String)} which is
+ * `ignite` by default.
+ * <p>
+ * As for Ignite pods, it's recommended to label them in such a way that the service will use the label in its selector
+ * configuration excluding endpoints of irrelevant Kubernetes pods running in parallel.
+ * <p>
+ * The address finder, in its turn, will call this service to retrieve Ignite pods IP addresses. The port will be
+ * set with {@link ReliableChannel#parseAddresses(String[])}. Make sure that all Ignite pods occupy a similar
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org