You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Jeff Jiao <je...@gmail.com> on 2018/01/03 06:40:01 UTC

Connection problem between client and server

Hi Ignite community,

we are encountering a connection problem, in our production environment, 
we have 3 Ignite servers with 500G+ of data each, and about 30 Ignite
clients running.
then we tried to start a simple Ignite client in different environment,
Citrix environment, it cannot connect to the existing servers. (firewalls
are open in both way for 47500-47509 and 47100-47109)


we did some investigating, 
1. if we start only one Ignite server node, this Ignite client from Citrix
can connect to it very fast, about 2-4 secs. (Server=1, Client=1)
2. then we start all of the 3 Ignite servers, it still can connect to
servers but took longer, about 20 secs. (Server=3, Client=1)
3. then we start 2 of our services which contains an Ignite client node,
then start this Ignite client in Citrix, it took 27-30 secs. (Server=3,
Client=3)


it looks like this connection establishment process depends on the amount of
Ignite nodes.
then we did some testing in our development environment and QA environment,
we used a tool to catch the packages transport from Ignite server to Ignite
client through port 47500 when they connect,
in DEV env, we have 1 server and 4 clients running, then start a client
node, the size of packages from server to this client is about 1MB.
in QA env, we have 3 servers and 12 clients running, then start a client
node, the size of packages from servers to this client is about 4MB.
The result also can prove the point.


we guess that between our Production env and Citrix env, when client try to
connect, there are too much infos need to be transport and reach some limit
or timeout...

so are we in the correct direction for debugging? do you have any ideas or
suggestions for this situation? or does Ignite has some configurations which
can control the information transportation when connect?


Thanks a lot,
Jeff



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Connection problem between client and server

Posted by Jeff Jiao <je...@gmail.com>.
Hi Denis,

We think the problem is when a client connect to an existing cluster, server
node will send the attributes of all the nodes in the cluster(Servers &
Clients) to that new client. it will be a lot of data here... 
We may also have problems for our network configuration, we are still
investigating.

For our system, it needs a lot of jars, we cannot avoid these infos...

Does Ignite dev team give any feedback for this?



Thanks,
Jeff



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Connection problem between client and server

Posted by Jeff Jiao <je...@gmail.com>.
Thank you very much Denis, we will give it a try and let you know, I think
this should solve our problem.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Connection problem between client and server

Posted by Denis Mekhanikov <dm...@gmail.com>.
Hi Jeff!

There is a workaround for the problem of huge discovery messages.
Try changing IgniteConfiguration.includeProperties
<https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/IgniteConfiguration.html#getIncludeProperties()>
parameter to an empty array. It will make Ignite filter out all environment
variables.
It should improve node connection time in your case.

Add the following lines to IgniteConfiguration in your XML:
<property name="includeProperties">
    <array></array>
</property>

Denis

ср, 10 янв. 2018 г. в 11:37, Denis Mekhanikov <dm...@gmail.com>:

> Hi Jeff.
>
> Looks like my letter wasn't noticed by the developer community.
>
> I sent a message to the dev list one more time:
> http://apache-ignite-developers.2346864.n4.nabble.com/Irrelevant-data-in-discovery-messages-td25927.html
>
> In the meanwhile make sure, that this is really the cause of the discovery
> process being slow. Try deploying nodes on the same environment, but
> without additional jar files on the classpath. Will it make discovery work
> faster?
>
> Denis
>
> ср, 10 янв. 2018 г. в 8:39, Jeff Jiao <je...@gmail.com>:
>
>> Hi Denis,
>>
>> Does Ignite dev team give any feedback for this?
>>
>> Thanks
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Connection problem between client and server

Posted by Denis Mekhanikov <dm...@gmail.com>.
Hi Jeff.

Looks like my letter wasn't noticed by the developer community.

I sent a message to the dev list one more time:
http://apache-ignite-developers.2346864.n4.nabble.com/Irrelevant-data-in-discovery-messages-td25927.html

In the meanwhile make sure, that this is really the cause of the discovery
process being slow. Try deploying nodes on the same environment, but
without additional jar files on the classpath. Will it make discovery work
faster?

Denis

ср, 10 янв. 2018 г. в 8:39, Jeff Jiao <je...@gmail.com>:

> Hi Denis,
>
> Does Ignite dev team give any feedback for this?
>
> Thanks
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Connection problem between client and server

Posted by Jeff Jiao <je...@gmail.com>.
Hi Denis,

Does Ignite dev team give any feedback for this?

Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Connection problem between client and server

Posted by Denis Mekhanikov <dm...@gmail.com>.
Jeff,

All node attributes, including environment variables and classpath are
transmitted in discovery messages.
So try to create an environment, where your app won't have this irrelevant
information.
To check, what is included into the node attributes, you can use the
following code:

ClusterNode locNode = ignite.cluster().localNode();

for (Map.Entry<String, Object> e : locNode.attributes().entrySet())
    System.out.println(e.getKey() + "=" + e.getValue());


> for the TCP window increasing, does Ignite has some attribute that we can configure?
or we may need to modify our production system.
Sorry, just ignore this recommendation. I figured out, that the size of
packages, that I see in your dump, depends on MTU network parameter. It's
better not to change it.

Denis

пт, 5 янв. 2018 г. в 6:18, Jeff Jiao <je...@gmail.com>:

> Hi Denis,
>
> Thanks for the analysis.
>
> Today we tried to reduce the Java class path(.../lib/*) when start a server
> node, but it still sends all the jar files(absolute path) to client.
>
> and we also find that, it looks like client node sends its info(path...) to
> server node, and then server node sends the client's info back to client
> again...
>
> for the TCP window increasing, does Ignite has some attribute that we can
> configure? or we may need to modify our production system.
>
> btw, we are using Ignite 2.3.0.
>
> Thanks,
> Jeff
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Connection problem between client and server

Posted by Jeff Jiao <je...@gmail.com>.
Hi Denis,

Thanks for the analysis.

Today we tried to reduce the Java class path(.../lib/*) when start a server
node, but it still sends all the jar files(absolute path) to client.

and we also find that, it looks like client node sends its info(path...) to
server node, and then server node sends the client's info back to client
again...

for the TCP window increasing, does Ignite has some attribute that we can
configure? or we may need to modify our production system.

btw, we are using Ignite 2.3.0.

Thanks,
Jeff



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Connection problem between client and server

Posted by Denis Mekhanikov <dm...@gmail.com>.
Jeff,

Your have indeed a lot of data sent over discovery. Almost all of it is
taken by node attributes. Try reducing Java classpath and PATH environment
variable. Also increasing of TCP window could help, because now a lot of
small messages are sent instead of a few big ones.

Cross-sending it to dev list:
Guys, turns out, we are sending a lot of irrelevant information in
discovery messages. Some messages contain *TcpDiscoveryNode* objects, which
in turn have such attributes like *PATH, java.class.path,
sun.boot.class.path, java.library.path, org.apache.ignite.jvm.args, *etc.
Some of these attributes may contain huge strings, that can sum up to
megabytes of data.

I think, we should filter-out such attributes, because they are not used
anywhere, but  make messages grow enormous and slow down discovery.

What do you think?

Denis

чт, 4 янв. 2018 г. в 10:42, Jeff Jiao <je...@gmail.com>:

> dev2.cap <
> http://apache-ignite-users.70518.x6.nabble.com/file/t757/dev2.cap>
>
> Hi Denis,
>
> the attachment is the packages that server transport to client when
> connect,
> we catched it using tool (Microsoft network monitor), can you please take a
> look and see what you can find?
>
> we only configured 12 caches in our Ignite, most of them have 3 or 4
> indexes, only few has 7 or 8 indexes configured...
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Connection problem between client and server

Posted by Denis Mekhanikov <dm...@gmail.com>.
Jeff,

Your have indeed a lot of data sent over discovery. Almost all of it is
taken by node attributes. Try reducing Java classpath and PATH environment
variable. Also increasing of TCP window could help, because now a lot of
small messages are sent instead of a few big ones.

Cross-sending it to dev list:
Guys, turns out, we are sending a lot of irrelevant information in
discovery messages. Some messages contain *TcpDiscoveryNode* objects, which
in turn have such attributes like *PATH, java.class.path,
sun.boot.class.path, java.library.path, org.apache.ignite.jvm.args, *etc.
Some of these attributes may contain huge strings, that can sum up to
megabytes of data.

I think, we should filter-out such attributes, because they are not used
anywhere, but  make messages grow enormous and slow down discovery.

What do you think?

Denis

чт, 4 янв. 2018 г. в 10:42, Jeff Jiao <je...@gmail.com>:

> dev2.cap <
> http://apache-ignite-users.70518.x6.nabble.com/file/t757/dev2.cap>
>
> Hi Denis,
>
> the attachment is the packages that server transport to client when
> connect,
> we catched it using tool (Microsoft network monitor), can you please take a
> look and see what you can find?
>
> we only configured 12 caches in our Ignite, most of them have 3 or 4
> indexes, only few has 7 or 8 indexes configured...
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Connection problem between client and server

Posted by Jeff Jiao <je...@gmail.com>.
dev2.cap <http://apache-ignite-users.70518.x6.nabble.com/file/t757/dev2.cap>  

Hi Denis,

the attachment is the packages that server transport to client when connect,
we catched it using tool (Microsoft network monitor), can you please take a
look and see what you can find?

we only configured 12 caches in our Ignite, most of them have 3 or 4
indexes, only few has 7 or 8 indexes configured...




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Connection problem between client and server

Posted by Denis Mekhanikov <dm...@gmail.com>.
Hi Jeff!

1 MB sound like a lot of data. Especially for discovery protocol. I didn't
check, what is the volume of discovery data normally, but I think, it
shouldn't be measured in megabytes. If you don't create thousands of
caches, of cause.
What work load do you perform on your cluster? How many caches do you
create? Can you provide node configuration files?

Node startup can be slow, if you have long ping between server nodes. Make
sure, that network between nodes is stable and has low latency.

Denis

ср, 3 янв. 2018 г. в 9:40, Jeff Jiao <je...@gmail.com>:

> Hi Ignite community,
>
> we are encountering a connection problem, in our production environment,
> we have 3 Ignite servers with 500G+ of data each, and about 30 Ignite
> clients running.
> then we tried to start a simple Ignite client in different environment,
> Citrix environment, it cannot connect to the existing servers. (firewalls
> are open in both way for 47500-47509 and 47100-47109)
>
>
> we did some investigating,
> 1. if we start only one Ignite server node, this Ignite client from Citrix
> can connect to it very fast, about 2-4 secs. (Server=1, Client=1)
> 2. then we start all of the 3 Ignite servers, it still can connect to
> servers but took longer, about 20 secs. (Server=3, Client=1)
> 3. then we start 2 of our services which contains an Ignite client node,
> then start this Ignite client in Citrix, it took 27-30 secs. (Server=3,
> Client=3)
>
>
> it looks like this connection establishment process depends on the amount
> of
> Ignite nodes.
> then we did some testing in our development environment and QA environment,
> we used a tool to catch the packages transport from Ignite server to Ignite
> client through port 47500 when they connect,
> in DEV env, we have 1 server and 4 clients running, then start a client
> node, the size of packages from server to this client is about 1MB.
> in QA env, we have 3 servers and 12 clients running, then start a client
> node, the size of packages from servers to this client is about 4MB.
> The result also can prove the point.
>
>
> we guess that between our Production env and Citrix env, when client try to
> connect, there are too much infos need to be transport and reach some limit
> or timeout...
>
> so are we in the correct direction for debugging? do you have any ideas or
> suggestions for this situation? or does Ignite has some configurations
> which
> can control the information transportation when connect?
>
>
> Thanks a lot,
> Jeff
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>