You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@geode.apache.org by Alberto Gomez <al...@est.tech> on 2019/07/23 12:48:02 UTC

About single-hop access to partitioned regions and number of client connections

Hi,

I have run some tests using two different clients, a Java one and a C++ 
one, both with single hop enabled, maxConnections not limited (set to 
-1), accessing a partitioned region, and have observed the following:

- If the amount of operations sent to the Geode cluster is not very 
high, the number of connections from the client to the servers remains 
constant and reasonably low (the number appears to be (number of threads 
in the client * number of servers in the Geode cluster)).

- If the amount of operations sent to the Geode cluster is very high 
then the behavior seems different between the Java and the C++ client:

     * With the Java client, the number of connections seems to be 
bounded to a maximum that seems to be (number of threads in the client * 
number of servers) although there is a continuous thrashing of 
connections (connections closed and opened continuously). According to 
my observations, the reason why connections are closed is because the 
server considers them idle.

     * With the C++ client, the number of connections seems to grow 
indefinitely and quite quickly. In this case, the amount of connections 
closed (if any) seems very small. This is a problem because sooner or 
later the client runs out of file descriptors.

I have some questions regarding the observed behavior:

- What fires the creation of new connections in the Java case? Is there 
a limit for the number of connections opened (there seems to be one)?

- In the Java case, what could be causing that some connections are idle 
given that the amount of operations sent is very high?

- In the C++ case, what fires the creation of new connections?

- In the C++, is it normal that the number of connections grows 
indefinitely?

- Is there a reason for the two client APIs to behave differently (in 
the Java one connections expire while in the C++ connections do not expire)?


Thanks in advance,

Alberto

Re: About single-hop access to partitioned regions and number of client connections

Posted by John Blum <jb...@pivotal.io>.

It is also worth mentioning that client connections can be controlled
(limited) on the server-side with the CacheServer.setMaxConnections(:int). [1]
I am not sure this would help on the client though if you have a large
number of Threads consuming connections from the Pool and attempting to
reach the server, but would (should) help control resource utilization on
the server if there are 1) a large number of busy clients 2) doing a bunch
of concurrent work.


[1]
https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/server/CacheServer.html#setMaxConnections-int-


On Tue, Jul 23, 2019 at 8:49 AM Jacob Barrett <jb...@pivotal.io> wrote:

> What version of Geode are you using?
>
> I ask because on develop there is a yet to be release fix to connection
> pooling for the Java client. The same fix has not been rolled out to the
> C++ client. For version 1.9 and older if you are using lots of threads in
> your client you will want to enable thread local connection pooling.
>
> Java:
> https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/client/PoolFactory.html#setThreadLocalConnections-boolean-
> C++:
> https://geode.apache.org/releases/latest/cppdocs/a00789.html#a62f037e2c91bc6869fafaad690040322
>
> In both C++ and Java clients the creation of new connections is caused by
> an exhausted pool. If you request a connection to a specific server, like
> when doing single hop, if there are no connections currently in the pool
> then a new connection is created. The apparent difference in Java and C++
> is likely caused by differences in lock contention in the pool. The Java
> client had serious lock contention that keep utilization low, which is
> fixed in develop (1.10). Thread local connections was an old attempt to
> address this issue but resulted in lots of connections being idle in a
> single-hop scenario.
>
> -Jake
>
> On Jul 23, 2019, at 7:39 AM, Alberto Gomez <al...@est.tech> wrote:
>
> Just a small correction on my original e-mail.
>
> When I said that "connections are closed because the server considered
> them idle" I meant that "connections are closed because the client
> library considered them idle".
>
> Best regards,
>
> Alberto
>
> On 23/7/19 14:48, Alberto Gomez wrote:
>
> Hi,
>
> I have run some tests using two different clients, a Java one and a C++
> one, both with single hop enabled, maxConnections not limited (set to
> -1), accessing a partitioned region, and have observed the following:
>
> - If the amount of operations sent to the Geode cluster is not very
> high, the number of connections from the client to the servers remains
> constant and reasonably low (the number appears to be (number of threads
> in the client * number of servers in the Geode cluster)).
>
> - If the amount of operations sent to the Geode cluster is very high
> then the behavior seems different between the Java and the C++ client:
>
>      * With the Java client, the number of connections seems to be
> bounded to a maximum that seems to be (number of threads in the client *
> number of servers) although there is a continuous thrashing of
> connections (connections closed and opened continuously). According to
> my observations, the reason why connections are closed is because the
> server considers them idle.
>
>      * With the C++ client, the number of connections seems to grow
> indefinitely and quite quickly. In this case, the amount of connections
> closed (if any) seems very small. This is a problem because sooner or
> later the client runs out of file descriptors.
>
> I have some questions regarding the observed behavior:
>
> - What fires the creation of new connections in the Java case? Is there
> a limit for the number of connections opened (there seems to be one)?
>
> - In the Java case, what could be causing that some connections are idle
> given that the amount of operations sent is very high?
>
> - In the C++ case, what fires the creation of new connections?
>
> - In the C++, is it normal that the number of connections grows
> indefinitely?
>
> - Is there a reason for the two client APIs to behave differently (in
> the Java one connections expire while in the C++ connections do not
> expire)?
>
>
> Thanks in advance,
>
> Alberto
>
>
>
>

-- 
-John
john.blum10101 (skype)

Re: About single-hop access to partitioned regions and number of client connections

Posted by Jacob Barrett <jb...@pivotal.io>.


> On Jul 25, 2019, at 6:51 AM, Alberto Gomez <al...@est.tech> wrote:
>> 
>> The C++ client connection pooling should behave a lot like the old client, so you probably want to enable thread local connections on the C++ client. It will eat more sockets but should behave better. Try and give feedback on how it addresses your file handles issue. Again load conditioning is probably fighting you if all you want is single-hop partition behavior. Check client connection idle timeout settings.
> I am using here the default value for idle timeout.
> 
> Using thread local connections makes the number of connections not go above a number (164 in a case in which I have two servers and 80 threads).
> 
> Regarding the closing of connections in the C++ client when idle, I found a possible bug in the code. A variable of type size_t is used to store possibly negative numbers in the ThinClientPoolDM::cleanStaleConnections which was preventing the closing of any idleConnection.
> 
> Once I have changed that variable to int, I see the closing of connections when the client is idle as well as a behavior similar to the Java client when there is high load (continous opening and closing of connections).
> 
> I will write JIRA about this problem.
> 
Awesome find! Thanks for digging in and writing up a JIRA.

Re: About single-hop access to partitioned regions and number of client connections

Posted by Alberto Gomez <al...@est.tech>.

Hi again,

Please, see my comments below:

On 23/7/19 18:37, Jacob Barrett wrote:

On Jul 23, 2019, at 9:29 AM, Alberto Gomez <al...@est.tech>> wrote:

I am using the today's version of the develop branch. According to your answer, I should have the fix applied on the Java client.

How many threads can be considered "lots"? My client uses around 40 threads and is limited to run on 4 CPUs.

That might a “lots” of threads in your scenario. In our geode-benchmarks we saw issues at thread counts more than 8 times the number of CPUs. We have not done benchmarking of the C++ client to see how it behaves under heavy thread load.

Another thing I have noticed is that connections in the C++ client are not closed when idle while in the Java client, if they are idle, are eventually closed. That would explain why in the Java client there is closing of connections while in the C++ there isn't. Nevertheless, I still do not see why connections may be closed in the Java client due to idleness with a high amount of operations being continually sent.

The clients do different things around connection management right now.
Java client should be only be closing idle connections after they reach their timeout. What is your idle timeout? They can also close during load conditioning. If you are only doing partitioned region operations you should just disable load conditioning. It has known issues in which it will fight single-hop connection pool balancing.
I am using the default value for idleTimeout which is 5 secs. If I disable load conditioning I do not see big changes.

The C++ client connection pooling should behave a lot like the old client, so you probably want to enable thread local connections on the C++ client. It will eat more sockets but should behave better. Try and give feedback on how it addresses your file handles issue. Again load conditioning is probably fighting you if all you want is single-hop partition behavior. Check client connection idle timeout settings.

I am using here the default value for idle timeout.

Using thread local connections makes the number of connections not go above a number (164 in a case in which I have two servers and 80 threads).

Regarding the closing of connections in the C++ client when idle, I found a possible bug in the code. A variable of type size_t is used to store possibly negative numbers in the ThinClientPoolDM::cleanStaleConnections which was preventing the closing of any idleConnection.

Once I have changed that variable to int, I see the closing of connections when the client is idle as well as a behavior similar to the Java client when there is high load (continous opening and closing of connections).

I will write JIRA about this problem.

-Jake
-Alberto

Re: About single-hop access to partitioned regions and number of client connections

Posted by Jacob Barrett <jb...@pivotal.io>.

> On Jul 23, 2019, at 9:29 AM, Alberto Gomez <al...@est.tech> wrote:
> I am using the today's version of the develop branch. According to your answer, I should have the fix applied on the Java client.
> 
> How many threads can be considered "lots"? My client uses around 40 threads and is limited to run on 4 CPUs.
> 

That might a “lots” of threads in your scenario. In our geode-benchmarks we saw issues at thread counts more than 8 times the number of CPUs. We have not done benchmarking of the C++ client to see how it behaves under heavy thread load.

> Another thing I have noticed is that connections in the C++ client are not closed when idle while in the Java client, if they are idle, are eventually closed. That would explain why in the Java client there is closing of connections while in the C++ there isn't. Nevertheless, I still do not see why connections may be closed in the Java client due to idleness with a high amount of operations being continually sent.
> 
The clients do different things around connection management right now.
Java client should be only be closing idle connections after they reach their timeout. What is your idle timeout? They can also close during load conditioning. If you are only doing partitioned region operations you should just disable load conditioning. It has known issues in which it will fight single-hop connection pool balancing. 

The C++ client connection pooling should behave a lot like the old client, so you probably want to enable thread local connections on the C++ client. It will eat more sockets but should behave better. Try and give feedback on how it addresses your file handles issue. Again load conditioning is probably fighting you if all you want is single-hop partition behavior. Check client connection idle timeout settings.

-Jake

Re: About single-hop access to partitioned regions and number of client connections

Posted by Alberto Gomez <al...@est.tech>.

Hi Jake,

Thanks for your answer.

I am using the today's version of the develop branch. According to your answer, I should have the fix applied on the Java client.

How many threads can be considered "lots"? My client uses around 40 threads and is limited to run on 4 CPUs.

Alberto

On 23/7/19 17:49, Jacob Barrett wrote:
What version of Geode are you using?

I ask because on develop there is a yet to be release fix to connection pooling for the Java client. The same fix has not been rolled out to the C++ client. For version 1.9 and older if you are using lots of threads in your client you will want to enable thread local connection pooling.

Java: https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/client/PoolFactory.html#setThreadLocalConnections-boolean-
C++: https://geode.apache.org/releases/latest/cppdocs/a00789.html#a62f037e2c91bc6869fafaad690040322

In both C++ and Java clients the creation of new connections is caused by an exhausted pool. If you request a connection to a specific server, like when doing single hop, if there are no connections currently in the pool then a new connection is created. The apparent difference in Java and C++ is likely caused by differences in lock contention in the pool. The Java client had serious lock contention that keep utilization low, which is fixed in develop (1.10). Thread local connections was an old attempt to address this issue but resulted in lots of connections being idle in a single-hop scenario.

-Jake

On Jul 23, 2019, at 7:39 AM, Alberto Gomez <al...@est.tech>> wrote:

Just a small correction on my original e-mail.

When I said that "connections are closed because the server considered
them idle" I meant that "connections are closed because the client
library considered them idle".

Best regards,

Alberto

On 23/7/19 14:48, Alberto Gomez wrote:
Hi,

I have run some tests using two different clients, a Java one and a C++
one, both with single hop enabled, maxConnections not limited (set to
-1), accessing a partitioned region, and have observed the following:

- If the amount of operations sent to the Geode cluster is not very
high, the number of connections from the client to the servers remains
constant and reasonably low (the number appears to be (number of threads
in the client * number of servers in the Geode cluster)).

- If the amount of operations sent to the Geode cluster is very high
then the behavior seems different between the Java and the C++ client:

* With the Java client, the number of connections seems to be
bounded to a maximum that seems to be (number of threads in the client *
number of servers) although there is a continuous thrashing of
connections (connections closed and opened continuously). According to
my observations, the reason why connections are closed is because the
server considers them idle.

* With the C++ client, the number of connections seems to grow
indefinitely and quite quickly. In this case, the amount of connections
closed (if any) seems very small. This is a problem because sooner or
later the client runs out of file descriptors.

I have some questions regarding the observed behavior:

- What fires the creation of new connections in the Java case? Is there
a limit for the number of connections opened (there seems to be one)?

- In the Java case, what could be causing that some connections are idle
given that the amount of operations sent is very high?

- In the C++ case, what fires the creation of new connections?

- In the C++, is it normal that the number of connections grows
indefinitely?

- Is there a reason for the two client APIs to behave differently (in
the Java one connections expire while in the C++ connections do not expire)?

Thanks in advance,

Alberto

Re: About single-hop access to partitioned regions and number of client connections

Posted by Jacob Barrett <jb...@pivotal.io>.

What version of Geode are you using?

I ask because on develop there is a yet to be release fix to connection pooling for the Java client. The same fix has not been rolled out to the C++ client. For version 1.9 and older if you are using lots of threads in your client you will want to enable thread local connection pooling.

Java: https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/client/PoolFactory.html#setThreadLocalConnections-boolean- <https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/client/PoolFactory.html#setThreadLocalConnections-boolean->
C++: https://geode.apache.org/releases/latest/cppdocs/a00789.html#a62f037e2c91bc6869fafaad690040322 <https://geode.apache.org/releases/latest/cppdocs/a00789.html#a62f037e2c91bc6869fafaad690040322>

In both C++ and Java clients the creation of new connections is caused by an exhausted pool. If you request a connection to a specific server, like when doing single hop, if there are no connections currently in the pool then a new connection is created. The apparent difference in Java and C++ is likely caused by differences in lock contention in the pool. The Java client had serious lock contention that keep utilization low, which is fixed in develop (1.10). Thread local connections was an old attempt to address this issue but resulted in lots of connections being idle in a single-hop scenario.

-Jake

> On Jul 23, 2019, at 7:39 AM, Alberto Gomez <al...@est.tech> wrote:
> 
> Just a small correction on my original e-mail.
> 
> When I said that "connections are closed because the server considered 
> them idle" I meant that "connections are closed because the client 
> library considered them idle".
> 
> Best regards,
> 
> Alberto
> 
> On 23/7/19 14:48, Alberto Gomez wrote:
>> Hi,
>> 
>> I have run some tests using two different clients, a Java one and a C++
>> one, both with single hop enabled, maxConnections not limited (set to
>> -1), accessing a partitioned region, and have observed the following:
>> 
>> - If the amount of operations sent to the Geode cluster is not very
>> high, the number of connections from the client to the servers remains
>> constant and reasonably low (the number appears to be (number of threads
>> in the client * number of servers in the Geode cluster)).
>> 
>> - If the amount of operations sent to the Geode cluster is very high
>> then the behavior seems different between the Java and the C++ client:
>> 
>>      * With the Java client, the number of connections seems to be
>> bounded to a maximum that seems to be (number of threads in the client *
>> number of servers) although there is a continuous thrashing of
>> connections (connections closed and opened continuously). According to
>> my observations, the reason why connections are closed is because the
>> server considers them idle.
>> 
>>      * With the C++ client, the number of connections seems to grow
>> indefinitely and quite quickly. In this case, the amount of connections
>> closed (if any) seems very small. This is a problem because sooner or
>> later the client runs out of file descriptors.
>> 
>> I have some questions regarding the observed behavior:
>> 
>> - What fires the creation of new connections in the Java case? Is there
>> a limit for the number of connections opened (there seems to be one)?
>> 
>> - In the Java case, what could be causing that some connections are idle
>> given that the amount of operations sent is very high?
>> 
>> - In the C++ case, what fires the creation of new connections?
>> 
>> - In the C++, is it normal that the number of connections grows
>> indefinitely?
>> 
>> - Is there a reason for the two client APIs to behave differently (in
>> the Java one connections expire while in the C++ connections do not expire)?
>> 
>> 
>> Thanks in advance,
>> 
>> Alberto
>> 
>>

Re: About single-hop access to partitioned regions and number of client connections

Posted by Alberto Gomez <al...@est.tech>.

Just a small correction on my original e-mail.

When I said that "connections are closed because the server considered 
them idle" I meant that "connections are closed because the client 
library considered them idle".

Best regards,

Alberto

On 23/7/19 14:48, Alberto Gomez wrote:
> Hi,
>
> I have run some tests using two different clients, a Java one and a C++
> one, both with single hop enabled, maxConnections not limited (set to
> -1), accessing a partitioned region, and have observed the following:
>
> - If the amount of operations sent to the Geode cluster is not very
> high, the number of connections from the client to the servers remains
> constant and reasonably low (the number appears to be (number of threads
> in the client * number of servers in the Geode cluster)).
>
> - If the amount of operations sent to the Geode cluster is very high
> then the behavior seems different between the Java and the C++ client:
>
>       * With the Java client, the number of connections seems to be
> bounded to a maximum that seems to be (number of threads in the client *
> number of servers) although there is a continuous thrashing of
> connections (connections closed and opened continuously). According to
> my observations, the reason why connections are closed is because the
> server considers them idle.
>
>       * With the C++ client, the number of connections seems to grow
> indefinitely and quite quickly. In this case, the amount of connections
> closed (if any) seems very small. This is a problem because sooner or
> later the client runs out of file descriptors.
>
> I have some questions regarding the observed behavior:
>
> - What fires the creation of new connections in the Java case? Is there
> a limit for the number of connections opened (there seems to be one)?
>
> - In the Java case, what could be causing that some connections are idle
> given that the amount of operations sent is very high?
>
> - In the C++ case, what fires the creation of new connections?
>
> - In the C++, is it normal that the number of connections grows
> indefinitely?
>
> - Is there a reason for the two client APIs to behave differently (in
> the Java one connections expire while in the C++ connections do not expire)?
>
>
> Thanks in advance,
>
> Alberto
>
>