You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geode.apache.org by Eric Pederson <er...@gmail.com> on 2015/10/14 23:29:08 UTC
Queue client is full - effects other clients?
Dear all:
I was tracking down an issue with a client that was receiving events late
(receipt time minutes later than sending time) and started spuriously
connecting/reconnecting to the server, ultimately losing some events.
In the server log I saw this during the same time period:
[warning 2015/10/13 16:28:48.551 EDT server-psclxfiprdsp104
<ServerConnection on port 37386 Thread 696> tid=0x66b] Client queue for
_gfe_non_durable_client_with_id_NYCWD28244(:loner):62826:929faf61_1_queue
client is full.
[info 2015/10/13 16:28:48.693 EDT server-psclxfiprdsp104 <ServerConnection
on port 37386 Thread 696> tid=0x66b] Resuming with processing puts ...
The client that was full was a different client from the one that was
getting its events late.
Is it possible that if a client is slow consuming its events (via CQ) and
its server-side queue becomes full that other clients will be effected? GC
activity looked normal on that server during the time period.
Thanks,
-- Eric
Re: Queue client is full - effects other clients?
Posted by Eric Pederson <er...@gmail.com>.
Thanks. That must have been it.
On Wednesday, October 14, 2015, Barry Oglesby <bo...@pivotal.io> wrote:
> Processing a client queue is asynchronous to the client operations, but
> adding to the queue is synchronous with them. So any client operation that
> needs to access the full queue will be blocked waiting for space to become
> available in it. That means any put operation from any client will block if
> the client with the full queue cares about that event. Depending on your
> client pool read-timeout, you could see client put operations timing out.
>
> The processing of other client queues shouldn't be affected by this since
> that processing is asynchronous from anything going on with the full queue.
> My guess would be those client queues are empty and only sporadically
> getting events added to them because a bunch of the putting threads are
> blocked.
>
> There is a property called remove-unresponsive-client (false by default)
> that you can use to kick out the full client so that other client
> operations are not affected.
>
> You can also raise the cache-server maximum-message-count, but that will
> probably just delay the issue.
>
>
> Barry Oglesby
> GemFire Advanced Customer Engineering (ACE)
> For immediate support please contact Pivotal Support at
> http://support.pivotal.io/
>
>
> On Wed, Oct 14, 2015 at 2:29 PM, Eric Pederson <ericacm@gmail.com
> <javascript:_e(%7B%7D,'cvml','ericacm@gmail.com');>> wrote:
>
>> Dear all:
>>
>> I was tracking down an issue with a client that was receiving events late
>> (receipt time minutes later than sending time) and started spuriously
>> connecting/reconnecting to the server, ultimately losing some events.
>>
>> In the server log I saw this during the same time period:
>>
>> [warning 2015/10/13 16:28:48.551 EDT server-psclxfiprdsp104
>> <ServerConnection on port 37386 Thread 696> tid=0x66b] Client queue for
>> _gfe_non_durable_client_with_id_NYCWD28244(:loner):62826:929faf61_1_queue
>> client is full.
>>
>>
>>
>> [info 2015/10/13 16:28:48.693 EDT server-psclxfiprdsp104
>> <ServerConnection on port 37386 Thread 696> tid=0x66b] Resuming with
>> processing puts ...
>>
>>
>> The client that was full was a different client from the one that was
>> getting its events late.
>>
>>
>> Is it possible that if a client is slow consuming its events (via CQ) and
>> its server-side queue becomes full that other clients will be effected? GC
>> activity looked normal on that server during the time period.
>>
>>
>> Thanks,
>>
>> -- Eric
>>
>
>
--
Sent from Gmail Mobile
Re: Queue client is full - effects other clients?
Posted by Vincent Ford <vf...@pivotal.io>.
Yes, as the client put operation will result in a server operation which
may get blocked by a client queue that is full on the server. The client
put needs to be completed on the server unless it is a local only operation
on client. Keep in mind puts, gets and queries don't go through the client
queue but a put can be effected as other clients with subscriptions will
need their queues updated by the put operation ( CQ, register interest) on
the server side.
*Vince Ford*
GemFire Sustenance Engineering
Beaverton, OR USA
503-533-3726 (office)
http://www.pivotal.io
Open Source Project Geode https://geode.incubator.apache.org/
<https://network.pivotal.io/products/project-geode>
On Thu, Oct 15, 2015 at 3:31 PM, Eric Pederson <er...@gmail.com> wrote:
> Hi Barry - I mean, if a CACHING_PROXY client does a put, will it block?
> That is, is the client's put synchronous from the client to the server to
> the full client queue?
>
>
> On Thursday, October 15, 2015, Barry Oglesby <bo...@pivotal.io> wrote:
>
>> The client configuration won't make any difference to the server
>> behavior. The server threads processing the client requests are the ones
>> that block on the put into the queue (or waiting for a replicate to do it).
>>
>> Barry Oglesby
>> GemFire Advanced Customer Engineering (ACE)
>> For immediate support please contact Pivotal Support at
>> http://support.pivotal.io/
>>
>>
>> On Thu, Oct 15, 2015 at 2:26 PM, Eric Pederson <er...@gmail.com> wrote:
>>
>>> If a client is using CACHING_PROXY for a region in this scenario would
>>> its puts block on the full client queue in this scenario?
>>>
>>> On Wednesday, October 14, 2015, Barry Oglesby <bo...@pivotal.io>
>>> wrote:
>>>
>>>> Processing a client queue is asynchronous to the client operations, but
>>>> adding to the queue is synchronous with them. So any client operation that
>>>> needs to access the full queue will be blocked waiting for space to become
>>>> available in it. That means any put operation from any client will block if
>>>> the client with the full queue cares about that event. Depending on your
>>>> client pool read-timeout, you could see client put operations timing out.
>>>>
>>>> The processing of other client queues shouldn't be affected by this
>>>> since that processing is asynchronous from anything going on with the full
>>>> queue. My guess would be those client queues are empty and only
>>>> sporadically getting events added to them because a bunch of the putting
>>>> threads are blocked.
>>>>
>>>> There is a property called remove-unresponsive-client (false by
>>>> default) that you can use to kick out the full client so that other client
>>>> operations are not affected.
>>>>
>>>> You can also raise the cache-server maximum-message-count, but that
>>>> will probably just delay the issue.
>>>>
>>>>
>>>> Barry Oglesby
>>>> GemFire Advanced Customer Engineering (ACE)
>>>> For immediate support please contact Pivotal Support at
>>>> http://support.pivotal.io/
>>>>
>>>>
>>>> On Wed, Oct 14, 2015 at 2:29 PM, Eric Pederson <er...@gmail.com>
>>>> wrote:
>>>>
>>>>> Dear all:
>>>>>
>>>>> I was tracking down an issue with a client that was receiving events
>>>>> late (receipt time minutes later than sending time) and started spuriously
>>>>> connecting/reconnecting to the server, ultimately losing some events.
>>>>>
>>>>> In the server log I saw this during the same time period:
>>>>>
>>>>> [warning 2015/10/13 16:28:48.551 EDT server-psclxfiprdsp104
>>>>> <ServerConnection on port 37386 Thread 696> tid=0x66b] Client queue for
>>>>> _gfe_non_durable_client_with_id_NYCWD28244(:loner):62826:929faf61_1_queue
>>>>> client is full.
>>>>>
>>>>>
>>>>>
>>>>> [info 2015/10/13 16:28:48.693 EDT server-psclxfiprdsp104
>>>>> <ServerConnection on port 37386 Thread 696> tid=0x66b] Resuming with
>>>>> processing puts ...
>>>>>
>>>>>
>>>>> The client that was full was a different client from the one that was
>>>>> getting its events late.
>>>>>
>>>>>
>>>>> Is it possible that if a client is slow consuming its events (via CQ)
>>>>> and its server-side queue becomes full that other clients will be
>>>>> effected? GC activity looked normal on that server during the time period.
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -- Eric
>>>>>
>>>>
>>>>
>>>
>>> --
>>> Sent from Gmail Mobile
>>>
>>
>>
>
> --
> Sent from Gmail Mobile
>
Re: Queue client is full - effects other clients?
Posted by Eric Pederson <er...@gmail.com>.
Hi Barry - I mean, if a CACHING_PROXY client does a put, will it block?
That is, is the client's put synchronous from the client to the server to
the full client queue?
On Thursday, October 15, 2015, Barry Oglesby <bo...@pivotal.io> wrote:
> The client configuration won't make any difference to the server behavior.
> The server threads processing the client requests are the ones that block
> on the put into the queue (or waiting for a replicate to do it).
>
> Barry Oglesby
> GemFire Advanced Customer Engineering (ACE)
> For immediate support please contact Pivotal Support at
> http://support.pivotal.io/
>
>
> On Thu, Oct 15, 2015 at 2:26 PM, Eric Pederson <ericacm@gmail.com
> <javascript:_e(%7B%7D,'cvml','ericacm@gmail.com');>> wrote:
>
>> If a client is using CACHING_PROXY for a region in this scenario would
>> its puts block on the full client queue in this scenario?
>>
>> On Wednesday, October 14, 2015, Barry Oglesby <boglesby@pivotal.io
>> <javascript:_e(%7B%7D,'cvml','boglesby@pivotal.io');>> wrote:
>>
>>> Processing a client queue is asynchronous to the client operations, but
>>> adding to the queue is synchronous with them. So any client operation that
>>> needs to access the full queue will be blocked waiting for space to become
>>> available in it. That means any put operation from any client will block if
>>> the client with the full queue cares about that event. Depending on your
>>> client pool read-timeout, you could see client put operations timing out.
>>>
>>> The processing of other client queues shouldn't be affected by this
>>> since that processing is asynchronous from anything going on with the full
>>> queue. My guess would be those client queues are empty and only
>>> sporadically getting events added to them because a bunch of the putting
>>> threads are blocked.
>>>
>>> There is a property called remove-unresponsive-client (false by default)
>>> that you can use to kick out the full client so that other client
>>> operations are not affected.
>>>
>>> You can also raise the cache-server maximum-message-count, but that will
>>> probably just delay the issue.
>>>
>>>
>>> Barry Oglesby
>>> GemFire Advanced Customer Engineering (ACE)
>>> For immediate support please contact Pivotal Support at
>>> http://support.pivotal.io/
>>>
>>>
>>> On Wed, Oct 14, 2015 at 2:29 PM, Eric Pederson <er...@gmail.com>
>>> wrote:
>>>
>>>> Dear all:
>>>>
>>>> I was tracking down an issue with a client that was receiving events
>>>> late (receipt time minutes later than sending time) and started spuriously
>>>> connecting/reconnecting to the server, ultimately losing some events.
>>>>
>>>> In the server log I saw this during the same time period:
>>>>
>>>> [warning 2015/10/13 16:28:48.551 EDT server-psclxfiprdsp104
>>>> <ServerConnection on port 37386 Thread 696> tid=0x66b] Client queue for
>>>> _gfe_non_durable_client_with_id_NYCWD28244(:loner):62826:929faf61_1_queue
>>>> client is full.
>>>>
>>>>
>>>>
>>>> [info 2015/10/13 16:28:48.693 EDT server-psclxfiprdsp104
>>>> <ServerConnection on port 37386 Thread 696> tid=0x66b] Resuming with
>>>> processing puts ...
>>>>
>>>>
>>>> The client that was full was a different client from the one that was
>>>> getting its events late.
>>>>
>>>>
>>>> Is it possible that if a client is slow consuming its events (via CQ)
>>>> and its server-side queue becomes full that other clients will be
>>>> effected? GC activity looked normal on that server during the time period.
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> -- Eric
>>>>
>>>
>>>
>>
>> --
>> Sent from Gmail Mobile
>>
>
>
--
Sent from Gmail Mobile
Re: Queue client is full - effects other clients?
Posted by Barry Oglesby <bo...@pivotal.io>.
The client configuration won't make any difference to the server behavior.
The server threads processing the client requests are the ones that block
on the put into the queue (or waiting for a replicate to do it).
Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at
http://support.pivotal.io/
On Thu, Oct 15, 2015 at 2:26 PM, Eric Pederson <er...@gmail.com> wrote:
> If a client is using CACHING_PROXY for a region in this scenario would its
> puts block on the full client queue in this scenario?
>
> On Wednesday, October 14, 2015, Barry Oglesby <bo...@pivotal.io> wrote:
>
>> Processing a client queue is asynchronous to the client operations, but
>> adding to the queue is synchronous with them. So any client operation that
>> needs to access the full queue will be blocked waiting for space to become
>> available in it. That means any put operation from any client will block if
>> the client with the full queue cares about that event. Depending on your
>> client pool read-timeout, you could see client put operations timing out.
>>
>> The processing of other client queues shouldn't be affected by this since
>> that processing is asynchronous from anything going on with the full queue.
>> My guess would be those client queues are empty and only sporadically
>> getting events added to them because a bunch of the putting threads are
>> blocked.
>>
>> There is a property called remove-unresponsive-client (false by default)
>> that you can use to kick out the full client so that other client
>> operations are not affected.
>>
>> You can also raise the cache-server maximum-message-count, but that will
>> probably just delay the issue.
>>
>>
>> Barry Oglesby
>> GemFire Advanced Customer Engineering (ACE)
>> For immediate support please contact Pivotal Support at
>> http://support.pivotal.io/
>>
>>
>> On Wed, Oct 14, 2015 at 2:29 PM, Eric Pederson <er...@gmail.com> wrote:
>>
>>> Dear all:
>>>
>>> I was tracking down an issue with a client that was receiving events
>>> late (receipt time minutes later than sending time) and started spuriously
>>> connecting/reconnecting to the server, ultimately losing some events.
>>>
>>> In the server log I saw this during the same time period:
>>>
>>> [warning 2015/10/13 16:28:48.551 EDT server-psclxfiprdsp104
>>> <ServerConnection on port 37386 Thread 696> tid=0x66b] Client queue for
>>> _gfe_non_durable_client_with_id_NYCWD28244(:loner):62826:929faf61_1_queue
>>> client is full.
>>>
>>>
>>>
>>> [info 2015/10/13 16:28:48.693 EDT server-psclxfiprdsp104
>>> <ServerConnection on port 37386 Thread 696> tid=0x66b] Resuming with
>>> processing puts ...
>>>
>>>
>>> The client that was full was a different client from the one that was
>>> getting its events late.
>>>
>>>
>>> Is it possible that if a client is slow consuming its events (via CQ)
>>> and its server-side queue becomes full that other clients will be
>>> effected? GC activity looked normal on that server during the time period.
>>>
>>>
>>> Thanks,
>>>
>>> -- Eric
>>>
>>
>>
>
> --
> Sent from Gmail Mobile
>
Re: Queue client is full - effects other clients?
Posted by Eric Pederson <er...@gmail.com>.
If a client is using CACHING_PROXY for a region in this scenario would its
puts block on the full client queue in this scenario?
On Wednesday, October 14, 2015, Barry Oglesby <bo...@pivotal.io> wrote:
> Processing a client queue is asynchronous to the client operations, but
> adding to the queue is synchronous with them. So any client operation that
> needs to access the full queue will be blocked waiting for space to become
> available in it. That means any put operation from any client will block if
> the client with the full queue cares about that event. Depending on your
> client pool read-timeout, you could see client put operations timing out.
>
> The processing of other client queues shouldn't be affected by this since
> that processing is asynchronous from anything going on with the full queue.
> My guess would be those client queues are empty and only sporadically
> getting events added to them because a bunch of the putting threads are
> blocked.
>
> There is a property called remove-unresponsive-client (false by default)
> that you can use to kick out the full client so that other client
> operations are not affected.
>
> You can also raise the cache-server maximum-message-count, but that will
> probably just delay the issue.
>
>
> Barry Oglesby
> GemFire Advanced Customer Engineering (ACE)
> For immediate support please contact Pivotal Support at
> http://support.pivotal.io/
>
>
> On Wed, Oct 14, 2015 at 2:29 PM, Eric Pederson <ericacm@gmail.com
> <javascript:_e(%7B%7D,'cvml','ericacm@gmail.com');>> wrote:
>
>> Dear all:
>>
>> I was tracking down an issue with a client that was receiving events late
>> (receipt time minutes later than sending time) and started spuriously
>> connecting/reconnecting to the server, ultimately losing some events.
>>
>> In the server log I saw this during the same time period:
>>
>> [warning 2015/10/13 16:28:48.551 EDT server-psclxfiprdsp104
>> <ServerConnection on port 37386 Thread 696> tid=0x66b] Client queue for
>> _gfe_non_durable_client_with_id_NYCWD28244(:loner):62826:929faf61_1_queue
>> client is full.
>>
>>
>>
>> [info 2015/10/13 16:28:48.693 EDT server-psclxfiprdsp104
>> <ServerConnection on port 37386 Thread 696> tid=0x66b] Resuming with
>> processing puts ...
>>
>>
>> The client that was full was a different client from the one that was
>> getting its events late.
>>
>>
>> Is it possible that if a client is slow consuming its events (via CQ) and
>> its server-side queue becomes full that other clients will be effected? GC
>> activity looked normal on that server during the time period.
>>
>>
>> Thanks,
>>
>> -- Eric
>>
>
>
--
Sent from Gmail Mobile
Re: Queue client is full - effects other clients?
Posted by Barry Oglesby <bo...@pivotal.io>.
Processing a client queue is asynchronous to the client operations, but
adding to the queue is synchronous with them. So any client operation that
needs to access the full queue will be blocked waiting for space to become
available in it. That means any put operation from any client will block if
the client with the full queue cares about that event. Depending on your
client pool read-timeout, you could see client put operations timing out.
The processing of other client queues shouldn't be affected by this since
that processing is asynchronous from anything going on with the full queue.
My guess would be those client queues are empty and only sporadically
getting events added to them because a bunch of the putting threads are
blocked.
There is a property called remove-unresponsive-client (false by default)
that you can use to kick out the full client so that other client
operations are not affected.
You can also raise the cache-server maximum-message-count, but that will
probably just delay the issue.
Barry Oglesby
GemFire Advanced Customer Engineering (ACE)
For immediate support please contact Pivotal Support at
http://support.pivotal.io/
On Wed, Oct 14, 2015 at 2:29 PM, Eric Pederson <er...@gmail.com> wrote:
> Dear all:
>
> I was tracking down an issue with a client that was receiving events late
> (receipt time minutes later than sending time) and started spuriously
> connecting/reconnecting to the server, ultimately losing some events.
>
> In the server log I saw this during the same time period:
>
> [warning 2015/10/13 16:28:48.551 EDT server-psclxfiprdsp104
> <ServerConnection on port 37386 Thread 696> tid=0x66b] Client queue for
> _gfe_non_durable_client_with_id_NYCWD28244(:loner):62826:929faf61_1_queue
> client is full.
>
>
>
> [info 2015/10/13 16:28:48.693 EDT server-psclxfiprdsp104 <ServerConnection
> on port 37386 Thread 696> tid=0x66b] Resuming with processing puts ...
>
>
> The client that was full was a different client from the one that was
> getting its events late.
>
>
> Is it possible that if a client is slow consuming its events (via CQ) and
> its server-side queue becomes full that other clients will be effected? GC
> activity looked normal on that server during the time period.
>
>
> Thanks,
>
> -- Eric
>
Re: Queue client is full - effects other clients?
Posted by Anilkumar Gingade <ag...@pivotal.io>.
When client queue start growing/full; it will impact the publisher
thread/client...
There is an option to kick-out the slow or unresponsive client...
Here is the info on it:
remove-unresponsive-client When this property is set to true, the primary
server drops unresponsive clients from all secondaries and itself. Clients
are deemed unresponsive when their messaging queues become full on the
server. While a client's queue is full, puts that would add to the queue
block on the server.
-Anil.
On Wed, Oct 14, 2015 at 2:29 PM, Eric Pederson <er...@gmail.com> wrote:
> Dear all:
>
> I was tracking down an issue with a client that was receiving events late
> (receipt time minutes later than sending time) and started spuriously
> connecting/reconnecting to the server, ultimately losing some events.
>
> In the server log I saw this during the same time period:
>
> [warning 2015/10/13 16:28:48.551 EDT server-psclxfiprdsp104
> <ServerConnection on port 37386 Thread 696> tid=0x66b] Client queue for
> _gfe_non_durable_client_with_id_NYCWD28244(:loner):62826:929faf61_1_queue
> client is full.
>
>
>
> [info 2015/10/13 16:28:48.693 EDT server-psclxfiprdsp104 <ServerConnection
> on port 37386 Thread 696> tid=0x66b] Resuming with processing puts ...
>
>
> The client that was full was a different client from the one that was
> getting its events late.
>
>
> Is it possible that if a client is slow consuming its events (via CQ) and
> its server-side queue becomes full that other clients will be effected? GC
> activity looked normal on that server during the time period.
>
>
> Thanks,
>
> -- Eric
>