Re: Errors with TCPCommunicationSpi when using zookeeper discovery

[Larry Mark <la...@principled.io>]

Wanted to close the loop on this, I am running in Kubernetes and the root
cause of this was a network policy blocking communications.

It seems as if in Ignite 2.6, with paired connections,  I need to have an
open communication path on 47100 between all the servers, between the
servers and the clients and between the clients.  Does this sound correct?
Are clients connecting to other clients?  Is this documented somewhere.

Thanks


On Thu, Jul 26, 2018 at 6:39 AM, Ilya Kasnacheev <il...@gmail.com>
wrote:

> Hello!
>
> As far as my understanding goes, ZookeeperClusterNode is not a Zookeeper
> daemon but a cluster node (i.e. Apache Ignite) which is managed by
> Zookeeper discovery. So it's natural that a connection will be initiated to
> such node.
>
> Modern TCP firewalls usually null route connection attempts to closed
> ports, hence you can not see communication problems for a long time and yet
> the connection won't be established.
>
> Regards,
>
> --
> Ilya Kasnacheev
>
> 2018-07-25 17:30 GMT+03:00 Larry Mark <la...@principled.io>:
>
>> The logs do not indicate any connectivity problem, unless I am missing
>> it, in which case please point it out to me.
>>
>> The messages seem to be getting through fine, but the server thinks there
>> is a connection which does not exist, so it rejects it.  This seems to
>> happen because the communication SPI has opened a connection to zookeeper.
>> Why would the communications SPI ( not discovery )  be initiating a
>> connection to zookeeper?
>> it seems like the connection to zookeeper is making an entry in the array
>> of GridCommunicationClient that is causing the communications SPI to think
>> a client is connected when it is not, and to reject the connections.
>>
>> If this is off base, please tell me what I am missing.
>>
>>
>>
>>
>>
>> On Wed, Jul 25, 2018 at 6:25 AM, Ilya Kasnacheev <
>> ilya.kasnacheev@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> This might happen when there's connectivity problems, i.e. when node A
>>> can connect to node B but not vice versa.
>>>
>>> You can also try increasing socketWriteTimeout on communicationSpi if
>>> communication connections die mid-flight.
>>>
>>>
>>> Regards,
>>>
>>> --
>>> Ilya Kasnacheev
>>>
>>> 2018-07-25 0:16 GMT+03:00 lmark58 <la...@principled.io>:
>>>
>>>> When using zoo keeper discovery I am getting intermittent errors with
>>>> clients
>>>> connecting.
>>>>
>>>> Scenario - I have one ignite server running, try to connect with 4
>>>> clients.
>>>>
>>>> For one of the connecting clients the socket between the client and
>>>> server
>>>> is never established and the log repeats the message
>>>> o.a.i.s.c.tcp.TcpCommunicationSpi - Received incoming connection from
>>>> remote
>>>> node while connecting to this node, rejecting
>>>>
>>>> The problem is intermittent, but whenever it happens, I see a log entry
>>>> that
>>>> the server is initiating a connection to zoo keeper.
>>>> o.a.i.s.c.tcp.TcpCommunicationSpi - Creating NIO client to node:
>>>> ZookeeperClusterNode [id=8b66ad54-0357-43a5-8d6e-0d11eda90b10,
>>>> addrs=[10.2.91.45, 127.0.0.1], order=2, loc=false, client=true]
>>>>
>>>> Attached is a section of the log that shows the clients connecting.  I
>>>> cannot post a full log to a public forum, but if the attached is not
>>>> enough,
>>>> once a contributor picks this up and replies I can send a longer log to
>>>> them
>>>> directly.
>>>>
>>>> Thanks
>>>> mini.log
>>>> <http://apache-ignite-users.70518.x6.nabble.com/file/t1285/mini.log>
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>
>>>
>>>
>>
>

Re: Errors with TCPCommunicationSpi when using zookeeper discovery

["ilya.kasnacheev" <il...@gmail.com>]

Hello!

I can imagine circumstances where clients will open connections to other
clients, such as starting caches on clients or deploying services on
clients.

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/