You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Alberto Gomez <al...@est.tech> on 2020/07/02 17:33:13 UTC
Question about gateway sender stopped and memory consumption
Hi,
We have observed that when a gateway sender is stopped in a site, all the events received while it is stopped are stored in the 'AbstractGatewaySender.tmpDroppedEvents' queue of the primary sender. The elements of this queue are not removed from this queue until the sender is started back again.
This behavior implies that if the gateway sender is stopped for a long time, there is a risk of heap exhaustion in the members hosting primary senders.
Under split brain situations, if lasting long enough, there could be heap exhaustion problems in servers due to the memory used by the gateway sender queues, even if overflow to disk is used -given that part of the event is always stored in memory.
For those situations we had thought about stopping gateway senders when the memory used by the gateway sender queues reached a certain memory threshold. But according to the above, stopping the gateway senders would only make things worse.
Would it make sense for the gateway sender not to store the received events in tmpDroppedEvents while it is stopped?
Any suggestion on how to approach the problem of heap exhaustion due to the growth of gateway sender queues in long lasting split brain situations?
Thanks in advance,
Alberto G.
Re: Question about gateway sender stopped and memory consumption
Posted by Alberto Gomez <al...@est.tech>.
Thanks for your answer, Kirk.
If we persist the unsent events in a persistent region then the memory consumed would not be as high but still it would not solve our problem with long lasting split brain as the persistent region would take some memory too to store those events even if they were overflown.
Ideally it should be backed up in a queue that does not use any memory.
Best regards,
Alberto
________________________________
From: Kirk Lund <kl...@apache.org>
Sent: Thursday, July 2, 2020 7:41 PM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: Question about gateway sender stopped and memory consumption
I would have expected unsent events to be stored in a queue that is backed
by a persistent region or something on disk. If that's not currently true,
then it seems like a good direction might be to make tmpDroppedEvents use a
durable queue of some sort that overflows to disk.
On Thu, Jul 2, 2020 at 10:33 AM Alberto Gomez <al...@est.tech>
wrote:
> Hi,
>
> We have observed that when a gateway sender is stopped in a site, all the
> events received while it is stopped are stored in the
> 'AbstractGatewaySender.tmpDroppedEvents' queue of the primary sender. The
> elements of this queue are not removed from this queue until the sender is
> started back again.
>
> This behavior implies that if the gateway sender is stopped for a long
> time, there is a risk of heap exhaustion in the members hosting primary
> senders.
>
> Under split brain situations, if lasting long enough, there could be heap
> exhaustion problems in servers due to the memory used by the gateway sender
> queues, even if overflow to disk is used -given that part of the event is
> always stored in memory.
> For those situations we had thought about stopping gateway senders when
> the memory used by the gateway sender queues reached a certain memory
> threshold. But according to the above, stopping the gateway senders would
> only make things worse.
>
> Would it make sense for the gateway sender not to store the received
> events in tmpDroppedEvents while it is stopped?
>
> Any suggestion on how to approach the problem of heap exhaustion due to
> the growth of gateway sender queues in long lasting split brain situations?
>
> Thanks in advance,
>
> Alberto G.
>
>
>
Re: Question about gateway sender stopped and memory consumption
Posted by Alberto Gomez <al...@est.tech>.
Thanks Juan!
I will check it.
Alberto
________________________________
From: Ju@N <ju...@gmail.com>
Sent: Thursday, July 2, 2020 7:46 PM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: Question about gateway sender stopped and memory consumption
I recall some discussion about this in the past, there even was an "RFC"
that never got implemented:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80452478.
Best regards.
On Thu, 2 Jul 2020 at 18:41, Kirk Lund <kl...@apache.org> wrote:
> I would have expected unsent events to be stored in a queue that is backed
> by a persistent region or something on disk. If that's not currently true,
> then it seems like a good direction might be to make tmpDroppedEvents use a
> durable queue of some sort that overflows to disk.
>
>
>
> On Thu, Jul 2, 2020 at 10:33 AM Alberto Gomez <al...@est.tech>
> wrote:
>
> > Hi,
> >
> > We have observed that when a gateway sender is stopped in a site, all the
> > events received while it is stopped are stored in the
> > 'AbstractGatewaySender.tmpDroppedEvents' queue of the primary sender. The
> > elements of this queue are not removed from this queue until the sender
> is
> > started back again.
> >
> > This behavior implies that if the gateway sender is stopped for a long
> > time, there is a risk of heap exhaustion in the members hosting primary
> > senders.
> >
> > Under split brain situations, if lasting long enough, there could be heap
> > exhaustion problems in servers due to the memory used by the gateway
> sender
> > queues, even if overflow to disk is used -given that part of the event is
> > always stored in memory.
> > For those situations we had thought about stopping gateway senders when
> > the memory used by the gateway sender queues reached a certain memory
> > threshold. But according to the above, stopping the gateway senders would
> > only make things worse.
> >
> > Would it make sense for the gateway sender not to store the received
> > events in tmpDroppedEvents while it is stopped?
> >
> > Any suggestion on how to approach the problem of heap exhaustion due to
> > the growth of gateway sender queues in long lasting split brain
> situations?
> >
> > Thanks in advance,
> >
> > Alberto G.
> >
> >
> >
>
--
Ju@N
Re: Question about gateway sender stopped and memory consumption
Posted by "Ju@N" <ju...@gmail.com>.
I recall some discussion about this in the past, there even was an "RFC"
that never got implemented:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80452478.
Best regards.
On Thu, 2 Jul 2020 at 18:41, Kirk Lund <kl...@apache.org> wrote:
> I would have expected unsent events to be stored in a queue that is backed
> by a persistent region or something on disk. If that's not currently true,
> then it seems like a good direction might be to make tmpDroppedEvents use a
> durable queue of some sort that overflows to disk.
>
>
>
> On Thu, Jul 2, 2020 at 10:33 AM Alberto Gomez <al...@est.tech>
> wrote:
>
> > Hi,
> >
> > We have observed that when a gateway sender is stopped in a site, all the
> > events received while it is stopped are stored in the
> > 'AbstractGatewaySender.tmpDroppedEvents' queue of the primary sender. The
> > elements of this queue are not removed from this queue until the sender
> is
> > started back again.
> >
> > This behavior implies that if the gateway sender is stopped for a long
> > time, there is a risk of heap exhaustion in the members hosting primary
> > senders.
> >
> > Under split brain situations, if lasting long enough, there could be heap
> > exhaustion problems in servers due to the memory used by the gateway
> sender
> > queues, even if overflow to disk is used -given that part of the event is
> > always stored in memory.
> > For those situations we had thought about stopping gateway senders when
> > the memory used by the gateway sender queues reached a certain memory
> > threshold. But according to the above, stopping the gateway senders would
> > only make things worse.
> >
> > Would it make sense for the gateway sender not to store the received
> > events in tmpDroppedEvents while it is stopped?
> >
> > Any suggestion on how to approach the problem of heap exhaustion due to
> > the growth of gateway sender queues in long lasting split brain
> situations?
> >
> > Thanks in advance,
> >
> > Alberto G.
> >
> >
> >
>
--
Ju@N
Re: Question about gateway sender stopped and memory consumption
Posted by Kirk Lund <kl...@apache.org>.
I would have expected unsent events to be stored in a queue that is backed
by a persistent region or something on disk. If that's not currently true,
then it seems like a good direction might be to make tmpDroppedEvents use a
durable queue of some sort that overflows to disk.
On Thu, Jul 2, 2020 at 10:33 AM Alberto Gomez <al...@est.tech>
wrote:
> Hi,
>
> We have observed that when a gateway sender is stopped in a site, all the
> events received while it is stopped are stored in the
> 'AbstractGatewaySender.tmpDroppedEvents' queue of the primary sender. The
> elements of this queue are not removed from this queue until the sender is
> started back again.
>
> This behavior implies that if the gateway sender is stopped for a long
> time, there is a risk of heap exhaustion in the members hosting primary
> senders.
>
> Under split brain situations, if lasting long enough, there could be heap
> exhaustion problems in servers due to the memory used by the gateway sender
> queues, even if overflow to disk is used -given that part of the event is
> always stored in memory.
> For those situations we had thought about stopping gateway senders when
> the memory used by the gateway sender queues reached a certain memory
> threshold. But according to the above, stopping the gateway senders would
> only make things worse.
>
> Would it make sense for the gateway sender not to store the received
> events in tmpDroppedEvents while it is stopped?
>
> Any suggestion on how to approach the problem of heap exhaustion due to
> the growth of gateway sender queues in long lasting split brain situations?
>
> Thanks in advance,
>
> Alberto G.
>
>
>