You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geode.apache.org by Olivier Mallassi <ol...@gmail.com> on 2019/01/11 16:44:28 UTC
Issues with register interest with the number of consumers
All
I am facing a kind of scaling mystery while using register interest.
Use-case is basic :
N publishers publishing into a geode cluster w/ M subscribers using
register_interest (regexp).
in our case,
- M can be up to 1000 process (not threads)
- Publisher throughput is about 50k requests per sec on 200k keys
- Volume is small (around 10GB)
We observe put (from publisher point of view) latency diverging with the
number of consumers, up to 10 sec (p99) and 5 sec (p50)
AFAIR, the key is evaluated on the put (sync).
Any ideas , customisation of the configuration you can advice?
Regards.
Oliv/
Re: Issues with register interest with the number of consumers
Posted by Olivier Mallassi <ol...@gmail.com>.
Apologies 28 cores each (not 8)
On Wed 16 Jan 2019 at 22:11, Olivier Mallassi <ol...@gmail.com>
wrote:
> All
>
> I do not have all the details here but
> - hardware is large (4 machines, 8 cores each). I do not have cpu metrics
> - it appears to be related to regexp evaluation. We tried “register by
> key” w/ interestresultpolicy.keys and get better results
>
> Now
> 1/ we are consuming too much memory. If I do the math I am consuming 90MB
> per subscription (no ha..) knowing I have 1000 subscriptions it meaans 70GB
> if I am not mistaken
> A priori the object size is small meaning that I should have a large
> number of keys in the subscription queue (that I do not understand) Am I
> right?
>
> 2/ when we start all the client processes (1000) it looks like we need 20
> minutes to have all subscriptions created. Any ideas of what we are missing?
>
> Any Help is welcomed, Even at the OS level. :)
>
> Regards
> Oliv/
>
>
> OSat 12 Jan 2019 at 11:05, Nabarun Nag <nn...@pivotal.io> wrote:
>
>> Hi,
>>
>> Could you please share more details about the objects and the hardware
>> specs.
>> Also, it will be really convenient if you share you git repo.
>> Regards
>> Naba
>>
>> On Sat, Jan 12, 2019 at 1:04 AM Olivier Mallassi <
>> olivier.mallassi@gmail.com> wrote:
>>
>>> Anil, Mike
>>>
>>> Thanks for your answers.AFAIK it is not correlated to GC. I will try to
>>> look in more details to the gfs metrics.
>>>
>>> Oliv/
>>>
>>>
>>> On Fri 11 Jan 2019 at 20:15, Michael Stolz <ms...@pivotal.io> wrote:
>>>
>>>> Olivier,
>>>>
>>>> Can you correlate anything like GC pauses with these slow deliveries?
>>>>
>>>> --
>>>> Mike Stolz
>>>> Principal Engineer, GemFire Product Lead
>>>> Mobile: +1-631-835-4771
>>>>
>>>>
>>>>
>>>> On Fri, Jan 11, 2019 at 1:36 PM Anilkumar Gingade <ag...@pivotal.io>
>>>> wrote:
>>>>
>>>>> Oliv,
>>>>>
>>>>> The subscription has cost associated with it; as interested need to be
>>>>> processed for all the clients; but its minimal looking for matching
>>>>> keys/regex.
>>>>> - The puts add the matched events to the client subscription queues on
>>>>> the server; do you see the client queues growing?
>>>>> - You may also need to see if there are any memory pressure on server.
>>>>> - If the clients are not consuming the events fast (slow clients) it
>>>>> may increase the server side queue size.
>>>>>
>>>>> -Anil.
>>>>>
>>>>>
>>>>> On Fri, Jan 11, 2019 at 8:56 AM Olivier Mallassi <
>>>>> olivier.mallassi@gmail.com> wrote:
>>>>>
>>>>>> sorry mistake : throughtput is 10 times less: 5k requests per sec.
>>>>>>
>>>>>> On Fri, Jan 11, 2019 at 5:44 PM Olivier Mallassi <
>>>>>> olivier.mallassi@gmail.com> wrote:
>>>>>>
>>>>>>> All
>>>>>>>
>>>>>>> I am facing a kind of scaling mystery while using register interest.
>>>>>>>
>>>>>>> Use-case is basic :
>>>>>>> N publishers publishing into a geode cluster w/ M subscribers using
>>>>>>> register_interest (regexp).
>>>>>>>
>>>>>>> in our case,
>>>>>>> - M can be up to 1000 process (not threads)
>>>>>>> - Publisher throughput is about 50k requests per sec on 200k keys
>>>>>>> - Volume is small (around 10GB)
>>>>>>>
>>>>>>> We observe put (from publisher point of view) latency diverging with
>>>>>>> the number of consumers, up to 10 sec (p99) and 5 sec (p50)
>>>>>>> AFAIR, the key is evaluated on the put (sync).
>>>>>>>
>>>>>>> Any ideas , customisation of the configuration you can advice?
>>>>>>>
>>>>>>> Regards.
>>>>>>>
>>>>>>> Oliv/
>>>>>>>
>>>>>>
Re: Issues with register interest with the number of consumers
Posted by Olivier Mallassi <ol...@gmail.com>.
All
I do not have all the details here but
- hardware is large (4 machines, 8 cores each). I do not have cpu metrics
- it appears to be related to regexp evaluation. We tried “register by key”
w/ interestresultpolicy.keys and get better results
Now
1/ we are consuming too much memory. If I do the math I am consuming 90MB
per subscription (no ha..) knowing I have 1000 subscriptions it meaans 70GB
if I am not mistaken
A priori the object size is small meaning that I should have a large number
of keys in the subscription queue (that I do not understand) Am I right?
2/ when we start all the client processes (1000) it looks like we need 20
minutes to have all subscriptions created. Any ideas of what we are missing?
Any Help is welcomed, Even at the OS level. :)
Regards
Oliv/
OSat 12 Jan 2019 at 11:05, Nabarun Nag <nn...@pivotal.io> wrote:
> Hi,
>
> Could you please share more details about the objects and the hardware
> specs.
> Also, it will be really convenient if you share you git repo.
> Regards
> Naba
>
> On Sat, Jan 12, 2019 at 1:04 AM Olivier Mallassi <
> olivier.mallassi@gmail.com> wrote:
>
>> Anil, Mike
>>
>> Thanks for your answers.AFAIK it is not correlated to GC. I will try to
>> look in more details to the gfs metrics.
>>
>> Oliv/
>>
>>
>> On Fri 11 Jan 2019 at 20:15, Michael Stolz <ms...@pivotal.io> wrote:
>>
>>> Olivier,
>>>
>>> Can you correlate anything like GC pauses with these slow deliveries?
>>>
>>> --
>>> Mike Stolz
>>> Principal Engineer, GemFire Product Lead
>>> Mobile: +1-631-835-4771
>>>
>>>
>>>
>>> On Fri, Jan 11, 2019 at 1:36 PM Anilkumar Gingade <ag...@pivotal.io>
>>> wrote:
>>>
>>>> Oliv,
>>>>
>>>> The subscription has cost associated with it; as interested need to be
>>>> processed for all the clients; but its minimal looking for matching
>>>> keys/regex.
>>>> - The puts add the matched events to the client subscription queues on
>>>> the server; do you see the client queues growing?
>>>> - You may also need to see if there are any memory pressure on server.
>>>> - If the clients are not consuming the events fast (slow clients) it
>>>> may increase the server side queue size.
>>>>
>>>> -Anil.
>>>>
>>>>
>>>> On Fri, Jan 11, 2019 at 8:56 AM Olivier Mallassi <
>>>> olivier.mallassi@gmail.com> wrote:
>>>>
>>>>> sorry mistake : throughtput is 10 times less: 5k requests per sec.
>>>>>
>>>>> On Fri, Jan 11, 2019 at 5:44 PM Olivier Mallassi <
>>>>> olivier.mallassi@gmail.com> wrote:
>>>>>
>>>>>> All
>>>>>>
>>>>>> I am facing a kind of scaling mystery while using register interest.
>>>>>>
>>>>>> Use-case is basic :
>>>>>> N publishers publishing into a geode cluster w/ M subscribers using
>>>>>> register_interest (regexp).
>>>>>>
>>>>>> in our case,
>>>>>> - M can be up to 1000 process (not threads)
>>>>>> - Publisher throughput is about 50k requests per sec on 200k keys
>>>>>> - Volume is small (around 10GB)
>>>>>>
>>>>>> We observe put (from publisher point of view) latency diverging with
>>>>>> the number of consumers, up to 10 sec (p99) and 5 sec (p50)
>>>>>> AFAIR, the key is evaluated on the put (sync).
>>>>>>
>>>>>> Any ideas , customisation of the configuration you can advice?
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>>> Oliv/
>>>>>>
>>>>>
Re: Issues with register interest with the number of consumers
Posted by Nabarun Nag <nn...@pivotal.io>.
Hi,
Could you please share more details about the objects and the hardware
specs.
Also, it will be really convenient if you share you git repo.
Regards
Naba
On Sat, Jan 12, 2019 at 1:04 AM Olivier Mallassi <ol...@gmail.com>
wrote:
> Anil, Mike
>
> Thanks for your answers.AFAIK it is not correlated to GC. I will try to
> look in more details to the gfs metrics.
>
> Oliv/
>
>
> On Fri 11 Jan 2019 at 20:15, Michael Stolz <ms...@pivotal.io> wrote:
>
>> Olivier,
>>
>> Can you correlate anything like GC pauses with these slow deliveries?
>>
>> --
>> Mike Stolz
>> Principal Engineer, GemFire Product Lead
>> Mobile: +1-631-835-4771
>>
>>
>>
>> On Fri, Jan 11, 2019 at 1:36 PM Anilkumar Gingade <ag...@pivotal.io>
>> wrote:
>>
>>> Oliv,
>>>
>>> The subscription has cost associated with it; as interested need to be
>>> processed for all the clients; but its minimal looking for matching
>>> keys/regex.
>>> - The puts add the matched events to the client subscription queues on
>>> the server; do you see the client queues growing?
>>> - You may also need to see if there are any memory pressure on server.
>>> - If the clients are not consuming the events fast (slow clients) it may
>>> increase the server side queue size.
>>>
>>> -Anil.
>>>
>>>
>>> On Fri, Jan 11, 2019 at 8:56 AM Olivier Mallassi <
>>> olivier.mallassi@gmail.com> wrote:
>>>
>>>> sorry mistake : throughtput is 10 times less: 5k requests per sec.
>>>>
>>>> On Fri, Jan 11, 2019 at 5:44 PM Olivier Mallassi <
>>>> olivier.mallassi@gmail.com> wrote:
>>>>
>>>>> All
>>>>>
>>>>> I am facing a kind of scaling mystery while using register interest.
>>>>>
>>>>> Use-case is basic :
>>>>> N publishers publishing into a geode cluster w/ M subscribers using
>>>>> register_interest (regexp).
>>>>>
>>>>> in our case,
>>>>> - M can be up to 1000 process (not threads)
>>>>> - Publisher throughput is about 50k requests per sec on 200k keys
>>>>> - Volume is small (around 10GB)
>>>>>
>>>>> We observe put (from publisher point of view) latency diverging with
>>>>> the number of consumers, up to 10 sec (p99) and 5 sec (p50)
>>>>> AFAIR, the key is evaluated on the put (sync).
>>>>>
>>>>> Any ideas , customisation of the configuration you can advice?
>>>>>
>>>>> Regards.
>>>>>
>>>>> Oliv/
>>>>>
>>>>
Re: Issues with register interest with the number of consumers
Posted by Olivier Mallassi <ol...@gmail.com>.
Anil, Mike
Thanks for your answers.AFAIK it is not correlated to GC. I will try to
look in more details to the gfs metrics.
Oliv/
On Fri 11 Jan 2019 at 20:15, Michael Stolz <ms...@pivotal.io> wrote:
> Olivier,
>
> Can you correlate anything like GC pauses with these slow deliveries?
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Lead
> Mobile: +1-631-835-4771
>
>
>
> On Fri, Jan 11, 2019 at 1:36 PM Anilkumar Gingade <ag...@pivotal.io>
> wrote:
>
>> Oliv,
>>
>> The subscription has cost associated with it; as interested need to be
>> processed for all the clients; but its minimal looking for matching
>> keys/regex.
>> - The puts add the matched events to the client subscription queues on
>> the server; do you see the client queues growing?
>> - You may also need to see if there are any memory pressure on server.
>> - If the clients are not consuming the events fast (slow clients) it may
>> increase the server side queue size.
>>
>> -Anil.
>>
>>
>> On Fri, Jan 11, 2019 at 8:56 AM Olivier Mallassi <
>> olivier.mallassi@gmail.com> wrote:
>>
>>> sorry mistake : throughtput is 10 times less: 5k requests per sec.
>>>
>>> On Fri, Jan 11, 2019 at 5:44 PM Olivier Mallassi <
>>> olivier.mallassi@gmail.com> wrote:
>>>
>>>> All
>>>>
>>>> I am facing a kind of scaling mystery while using register interest.
>>>>
>>>> Use-case is basic :
>>>> N publishers publishing into a geode cluster w/ M subscribers using
>>>> register_interest (regexp).
>>>>
>>>> in our case,
>>>> - M can be up to 1000 process (not threads)
>>>> - Publisher throughput is about 50k requests per sec on 200k keys
>>>> - Volume is small (around 10GB)
>>>>
>>>> We observe put (from publisher point of view) latency diverging with
>>>> the number of consumers, up to 10 sec (p99) and 5 sec (p50)
>>>> AFAIR, the key is evaluated on the put (sync).
>>>>
>>>> Any ideas , customisation of the configuration you can advice?
>>>>
>>>> Regards.
>>>>
>>>> Oliv/
>>>>
>>>
Re: Issues with register interest with the number of consumers
Posted by Michael Stolz <ms...@pivotal.io>.
Olivier,
Can you correlate anything like GC pauses with these slow deliveries?
--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
On Fri, Jan 11, 2019 at 1:36 PM Anilkumar Gingade <ag...@pivotal.io>
wrote:
> Oliv,
>
> The subscription has cost associated with it; as interested need to be
> processed for all the clients; but its minimal looking for matching
> keys/regex.
> - The puts add the matched events to the client subscription queues on the
> server; do you see the client queues growing?
> - You may also need to see if there are any memory pressure on server.
> - If the clients are not consuming the events fast (slow clients) it may
> increase the server side queue size.
>
> -Anil.
>
>
> On Fri, Jan 11, 2019 at 8:56 AM Olivier Mallassi <
> olivier.mallassi@gmail.com> wrote:
>
>> sorry mistake : throughtput is 10 times less: 5k requests per sec.
>>
>> On Fri, Jan 11, 2019 at 5:44 PM Olivier Mallassi <
>> olivier.mallassi@gmail.com> wrote:
>>
>>> All
>>>
>>> I am facing a kind of scaling mystery while using register interest.
>>>
>>> Use-case is basic :
>>> N publishers publishing into a geode cluster w/ M subscribers using
>>> register_interest (regexp).
>>>
>>> in our case,
>>> - M can be up to 1000 process (not threads)
>>> - Publisher throughput is about 50k requests per sec on 200k keys
>>> - Volume is small (around 10GB)
>>>
>>> We observe put (from publisher point of view) latency diverging with the
>>> number of consumers, up to 10 sec (p99) and 5 sec (p50)
>>> AFAIR, the key is evaluated on the put (sync).
>>>
>>> Any ideas , customisation of the configuration you can advice?
>>>
>>> Regards.
>>>
>>> Oliv/
>>>
>>
Re: Issues with register interest with the number of consumers
Posted by Anilkumar Gingade <ag...@pivotal.io>.
Oliv,
The subscription has cost associated with it; as interested need to be
processed for all the clients; but its minimal looking for matching
keys/regex.
- The puts add the matched events to the client subscription queues on the
server; do you see the client queues growing?
- You may also need to see if there are any memory pressure on server.
- If the clients are not consuming the events fast (slow clients) it may
increase the server side queue size.
-Anil.
On Fri, Jan 11, 2019 at 8:56 AM Olivier Mallassi <ol...@gmail.com>
wrote:
> sorry mistake : throughtput is 10 times less: 5k requests per sec.
>
> On Fri, Jan 11, 2019 at 5:44 PM Olivier Mallassi <
> olivier.mallassi@gmail.com> wrote:
>
>> All
>>
>> I am facing a kind of scaling mystery while using register interest.
>>
>> Use-case is basic :
>> N publishers publishing into a geode cluster w/ M subscribers using
>> register_interest (regexp).
>>
>> in our case,
>> - M can be up to 1000 process (not threads)
>> - Publisher throughput is about 50k requests per sec on 200k keys
>> - Volume is small (around 10GB)
>>
>> We observe put (from publisher point of view) latency diverging with the
>> number of consumers, up to 10 sec (p99) and 5 sec (p50)
>> AFAIR, the key is evaluated on the put (sync).
>>
>> Any ideas , customisation of the configuration you can advice?
>>
>> Regards.
>>
>> Oliv/
>>
>
Re: Issues with register interest with the number of consumers
Posted by Olivier Mallassi <ol...@gmail.com>.
sorry mistake : throughtput is 10 times less: 5k requests per sec.
On Fri, Jan 11, 2019 at 5:44 PM Olivier Mallassi <ol...@gmail.com>
wrote:
> All
>
> I am facing a kind of scaling mystery while using register interest.
>
> Use-case is basic :
> N publishers publishing into a geode cluster w/ M subscribers using
> register_interest (regexp).
>
> in our case,
> - M can be up to 1000 process (not threads)
> - Publisher throughput is about 50k requests per sec on 200k keys
> - Volume is small (around 10GB)
>
> We observe put (from publisher point of view) latency diverging with the
> number of consumers, up to 10 sec (p99) and 5 sec (p50)
> AFAIR, the key is evaluated on the put (sync).
>
> Any ideas , customisation of the configuration you can advice?
>
> Regards.
>
> Oliv/
>