You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Anton Mushin <An...@epam.com> on 2017/10/02 09:13:05 UTC
Trouble to connect to ignite cluster on kubernetes
Hi everyone!
Could you say what way for correct connect to ignite cluster on kubernetes?
I'm using ignite version 2.2.0 and try connect to cluster from my local machine with next configuration
private IgniteConfiguration getConfig(){
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder(false);
ipFinder.setAddresses(Arrays.asList("kuber_external_host_addr", " kuber_external_host_addr:forwarded_port"));
TcpCommunicationSpi commSpi=new TcpCommunicationSpi();
commSpi.setSharedMemoryPort(-1);
TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
tcpDiscoverySpi.setIpFinder(ipFinder);
tcpDiscoverySpi.setNetworkTimeout(TcpDiscoverySpi.DFLT_NETWORK_TIMEOUT *3);
return new IgniteConfiguration()
.setDiscoverySpi(tcpDiscoverySpi)
.setIgniteInstanceName(UUID.randomUUID().toString())
.setCommunicationSpi(commSpi);
}
I use default configuration for nods on cluster.
Then I'm trying connect to cluster I get error on my local machine:
[SEVERE][main][TcpDiscoverySpi] Exception on direct send: Connection refused: connect
java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1376)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1339)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1159)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1006)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:851)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:358)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1834)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:842)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1786)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:978)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1896)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1648)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1076)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:596)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:520)
at org.apache.ignite.Ignition.start(Ignition.java:322)
And in this moment in logs of cluster I see information about connect new server node to cluster group and instantaneous it disconnect.
RE: Trouble to connect to ignite cluster on kubernetes
Posted by Anton Mushin <An...@epam.com>.
Thanks for your replies!
This suggestion doesn't suit me, if I correctly understand what do hostNetwork=true, then I should have one ignite pod in one kubernetes node.
In one real kubernetes node I have a few ignite nodes in different namespaces, and I use ignite like as kubernetes service inside namespaces.
From: Denis Magda [mailto:dmagda@apache.org]
Sent: Monday, October 02, 2017 10:18 PM
To: user@ignite.apache.org
Cc: Anton Mushin <An...@epam.com>
Subject: Re: Trouble to connect to ignite cluster on kubernetes
Anton,
In general, the application has to be in the same network where the cluster nodes are. Otherwise you might face various network related issues due to the specificity on how Kubernetes manages the network.
There is a ticket to cover your use case. A new IP finder should be created for the nodes connecting outside.
https://issues.apache.org/jira/browse/IGNITE-4161
Talking to kubernetes experts I learnt that this might not be needed if you do the following:
· Set hostNetwork=true in Ignite pod's YAML configuration so that the nodes that will be connecting from outside can establish TCP/IP connections with containerized Ignite nodes (pods).
· Use the same TcpDiscoveryKubernetesIpFinder for the nodes that will be outside of Kubernetes environment.
—
Denis
On Oct 2, 2017, at 5:45 AM, Nikolai Tikhonov <nt...@apache.org>> wrote:
Hi Anton!
For work Apache Ignite cluster requires that all nodes are able to communicate to each other directly. If I understood correctly kubernete starts up internal network interfaces and this environment looks similar that Apache Ignite works behind proxy. In this case you need to configure BasicAddressResolver which will map internal IP adresses on external host.
On Mon, Oct 2, 2017 at 12:13 PM, Anton Mushin <An...@epam.com>> wrote:
Hi everyone!
Could you say what way for correct connect to ignite cluster on kubernetes?
I'm using ignite version 2.2.0 and try connect to cluster from my local machine with next configuration
private IgniteConfiguration getConfig(){
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder(false);
ipFinder.setAddresses(Arrays.asList("kuber_external_host_addr", " kuber_external_host_addr:forwarded_port"));
TcpCommunicationSpi commSpi=new TcpCommunicationSpi();
commSpi.setSharedMemoryPort(-1);
TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
tcpDiscoverySpi.setIpFinder(ipFinder);
tcpDiscoverySpi.setNetworkTimeout(TcpDiscoverySpi.DFLT_NETWORK_TIMEOUT *3);
return new IgniteConfiguration()
.setDiscoverySpi(tcpDiscoverySpi)
.setIgniteInstanceName(UUID.randomUUID().toString())
.setCommunicationSpi(commSpi);
}
I use default configuration for nods on cluster.
Then I’m trying connect to cluster I get error on my local machine:
[SEVERE][main][TcpDiscoverySpi] Exception on direct send: Connection refused: connect
java.net.ConnectException: Connection refused: connect
at java.net<http://java.net/>.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net<http://java.net/>.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net<http://java.net/>.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net<http://java.net/>.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net<http://java.net/>.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1376)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1339)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1159)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1006)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:851)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:358)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1834)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:842)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1786)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:978)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1896)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1648)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1076)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:596)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:520)
at org.apache.ignite.Ignition.start(Ignition.java:322)
And in this moment in logs of cluster I see information about connect new server node to cluster group and instantaneous it disconnect.
Re: Trouble to connect to ignite cluster on kubernetes
Posted by Denis Magda <dm...@apache.org>.
Anton,
In general, the application has to be in the same network where the cluster nodes are. Otherwise you might face various network related issues due to the specificity on how Kubernetes manages the network.
There is a ticket to cover your use case. A new IP finder should be created for the nodes connecting outside.
https://issues.apache.org/jira/browse/IGNITE-4161 <https://issues.apache.org/jira/browse/IGNITE-4161>
Talking to kubernetes experts I learnt that this might not be needed if you do the following:
Set hostNetwork=true in Ignite pod's YAML configuration so that the nodes that will be connecting from outside can establish TCP/IP connections with containerized Ignite nodes (pods).
Use the same TcpDiscoveryKubernetesIpFinder for the nodes that will be outside of Kubernetes environment.
—
Denis
> On Oct 2, 2017, at 5:45 AM, Nikolai Tikhonov <nt...@apache.org> wrote:
>
> Hi Anton!
>
> For work Apache Ignite cluster requires that all nodes are able to communicate to each other directly. If I understood correctly kubernete starts up internal network interfaces and this environment looks similar that Apache Ignite works behind proxy. In this case you need to configure BasicAddressResolver which will map internal IP adresses on external host.
>
> On Mon, Oct 2, 2017 at 12:13 PM, Anton Mushin <Anton_Mushin@epam.com <ma...@epam.com>> wrote:
> Hi everyone!
>
>
>
> Could you say what way for correct connect to ignite cluster on kubernetes?
>
> I'm using ignite version 2.2.0 and try connect to cluster from my local machine with next configuration
>
>
>
> private IgniteConfiguration getConfig(){
>
> TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder(false);
>
> ipFinder.setAddresses(Arrays.asList("kuber_external_host_addr", " kuber_external_host_addr:forwarded_port"));
>
>
>
> TcpCommunicationSpi commSpi=new TcpCommunicationSpi();
>
> commSpi.setSharedMemoryPort(-1);
>
>
>
> TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
>
> tcpDiscoverySpi.setIpFinder(ipFinder);
>
> tcpDiscoverySpi.setNetworkTimeout(TcpDiscoverySpi.DFLT_NETWORK_TIMEOUT *3);
>
>
>
> return new IgniteConfiguration()
>
> .setDiscoverySpi(tcpDiscoverySpi)
>
> .setIgniteInstanceName(UUID.randomUUID().toString())
>
> .setCommunicationSpi(commSpi);
>
> }
>
>
>
> I use default configuration for nods on cluster.
>
> Then I’m trying connect to cluster I get error on my local machine:
>
>
>
> [SEVERE][main][TcpDiscoverySpi] Exception on direct send: Connection refused: connect
>
> java.net.ConnectException: Connection refused: connect
>
> at java.net <http://java.net/>.DualStackPlainSocketImpl.waitForConnect(Native Method)
>
> at java.net <http://java.net/>.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
>
> at java.net <http://java.net/>.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>
> at java.net <http://java.net/>.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
> at java.net <http://java.net/>.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:589)
>
> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1376)
>
> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1339)
>
> at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1159)
>
> at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1006)
>
> at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:851)
>
> at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:358)
>
> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1834)
>
> at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
>
> at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:842)
>
> at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1786)
>
> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:978)
>
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1896)
>
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1648)
>
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1076)
>
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:596)
>
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:520)
>
> at org.apache.ignite.Ignition.start(Ignition.java:322)
>
>
>
> And in this moment in logs of cluster I see information about connect new server node to cluster group and instantaneous it disconnect.
>
>
>
>
Re: Trouble to connect to ignite cluster on kubernetes
Posted by Nikolai Tikhonov <nt...@apache.org>.
Hi Anton!
For work Apache Ignite cluster requires that all nodes are able to
communicate to each other directly. If I understood correctly kubernete
starts up internal network interfaces and this environment looks similar
that Apache Ignite works behind proxy. In this case you need to configure
BasicAddressResolver which will map internal IP adresses on external host.
On Mon, Oct 2, 2017 at 12:13 PM, Anton Mushin <An...@epam.com> wrote:
> Hi everyone!
>
>
>
> Could you say what way for correct connect to ignite cluster on kubernetes?
>
> I'm using ignite version 2.2.0 and try connect to cluster from my local
> machine with next configuration
>
>
>
> private IgniteConfiguration getConfig(){
>
> TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder(false);
>
> ipFinder.setAddresses(Arrays.asList("kuber_external_host_addr", "
> kuber_external_host_addr:forwarded_port"));
>
>
>
> TcpCommunicationSpi commSpi=new TcpCommunicationSpi();
>
> commSpi.setSharedMemoryPort(-1);
>
>
>
> TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
>
> tcpDiscoverySpi.setIpFinder(ipFinder);
>
> tcpDiscoverySpi.setNetworkTimeout(TcpDiscoverySpi.DFLT_NETWORK_TIMEOUT
> *3);
>
>
>
> return new IgniteConfiguration()
>
> .setDiscoverySpi(tcpDiscoverySpi)
>
> .setIgniteInstanceName(UUID.randomUUID().toString())
>
> .setCommunicationSpi(commSpi);
>
> }
>
>
>
> I use default configuration for nods on cluster.
>
> Then I’m trying connect to cluster I get error on my local machine:
>
>
>
> [SEVERE][main][TcpDiscoverySpi] Exception on direct send: Connection
> refused: connect
>
> java.net.ConnectException: Connection refused: connect
>
> at java.net.DualStackPlainSocketImpl.waitForConnect(Native
> Method)
>
> at java.net.DualStackPlainSocketI
> mpl.socketConnect(DualStackPlainSocketImpl.java:85)
>
> at java.net.AbstractPlainSocketIm
> pl.doConnect(AbstractPlainSocketImpl.java:350)
>
> at java.net.AbstractPlainSocketIm
> pl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
> at java.net.AbstractPlainSocketIm
> pl.connect(AbstractPlainSocketImpl.java:188)
>
> at java.net.PlainSocketImpl.conne
> ct(PlainSocketImpl.java:172)
>
> at java.net.SocksSocketImpl.conne
> ct(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:589)
>
> at org.apache.ignite.spi.discover
> y.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1376)
>
> at org.apache.ignite.spi.discover
> y.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1339)
>
> at org.apache.ignite.spi.discover
> y.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1159)
>
> at org.apache.ignite.spi.discover
> y.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1006)
>
> at org.apache.ignite.spi.discover
> y.tcp.ServerImpl.joinTopology(ServerImpl.java:851)
>
> at org.apache.ignite.spi.discover
> y.tcp.ServerImpl.spiStart(ServerImpl.java:358)
>
> at org.apache.ignite.spi.discover
> y.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1834)
>
> at org.apache.ignite.internal.man
> agers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
>
> at org.apache.ignite.internal.man
> agers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:842)
>
> at org.apache.ignite.internal.Ign
> iteKernal.startManager(IgniteKernal.java:1786)
>
> at org.apache.ignite.internal.Ign
> iteKernal.start(IgniteKernal.java:978)
>
> at org.apache.ignite.internal.Ign
> itionEx$IgniteNamedInstance.start0(IgnitionEx.java:1896)
>
> at org.apache.ignite.internal.Ign
> itionEx$IgniteNamedInstance.start(IgnitionEx.java:1648)
>
> at org.apache.ignite.internal.Ign
> itionEx.start0(IgnitionEx.java:1076)
>
> at org.apache.ignite.internal.Ign
> itionEx.start(IgnitionEx.java:596)
>
> at org.apache.ignite.internal.Ign
> itionEx.start(IgnitionEx.java:520)
>
> at org.apache.ignite.Ignition.start(Ignition.java:322)
>
>
>
> And in this moment in logs of cluster I see information about connect new
> server node to cluster group and instantaneous it disconnect.
>
>
>