You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Mahesh Renduchintala <ma...@aline-consulting.com> on 2019/08/02 03:16:21 UTC
Re: Ignite 2.7.0: Ignite client:: memory leak
Denis,
Thanks. Meanwhile, we made some progress...the via workaround seems to use these two flags in the thick client config
-DIGNITE_SKIP_PARTITION_SIZE_VALIDATION -DIGNITE_EXCHANGE_HISTORY_SIZE=8
So far, we haven't seen clients going OOM for about 24 hours (still watching)
Based on the logs, you could see there is a partition map exchanged among the clients and feels like messages continuously get stored on the clients causing this problem.
Re: Ignite 2.7.0: Ignite client:: memory leak
Posted by Mahesh Renduchintala <ma...@aline-consulting.com>.
We observed the same memory leak issue on 2.7.6 as well. Deleting tables continuously from a thick client causes out of memory exceptions in other thick clients. Can you please recheck? We badly need a workaround for this issue.
This is probably related to another issue that is discussed on another thread
http://apache-ignite-users.70518.x6.nabble.com/Nodes-are-restarting-when-i-try-to-drop-a-table-created-with-persistence-enabled-td27897.html
Re: Ignite 2.7.0: Ignite client:: memory leak
Posted by Mahesh Renduchintala <ma...@aline-consulting.com>.
Dennis, Thanks for the update. Will wait for the release.
Re: Ignite 2.7.0: Ignite client:: memory leak
Posted by Denis Magda <dm...@apache.org>.
As an update, this issue has been included into Ignite 2.7.6 release
scheduled for this month.
-
Denis
On Tue, Aug 6, 2019 at 12:37 PM Ilya Kasnacheev <il...@gmail.com>
wrote:
> Hello!
>
> We did not change it since it should never be a problem anymore. The
> rationale is that I can't prove that some history size value will work and
> cause no further issues.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 6 авг. 2019 г. в 22:25, Denis Magda <dm...@apache.org>:
>
>> Andrey, Ilya, as part of IGNITE-11767, have we set the history size to 0
>> for the clients? If haven't what is a rationale?
>>
>> -
>> Denis
>>
>>
>> On Tue, Aug 6, 2019 at 5:56 AM Andrei Aleksandrov <
>> aealexsandrov@gmail.com> wrote:
>>
>>> Hi Mahesh,
>>>
>>> Yes, it's a problem related to IGNITE_EXCHANGE_HISTORY_SIZE. Ignite
>>> stored the data for the last 1000 exchanges.
>>>
>>> It generally can be required for the case when the coordinator was
>>> changed and new coordinator required to load last exchange history.
>>>
>>> Exist two problems here:
>>>
>>> 1)Client nodes can't be a coordinator. So there is no reason to store
>>> 1000 entries there. Will be better to set this option to some small
>>> value or zero for client nodes.
>>> 2)Server nodes also don't require 1000 entries. The required number of
>>> exchange history can depend on the number of server nodes. I suggest
>>> change the default value to small value.
>>>
>>> Here is the ticket related to this problem:
>>>
>>> https://issues.apache.org/jira/browse/IGNITE-11767
>>>
>>> It fixed and should be available in Ignite 2.8 where these exchanges
>>> will take less memory.
>>>
>>> BR,
>>> Andrei
>>>
>>> On 2019/08/03 01:09:19, Mahesh Renduchintala <m....@aline-consulting.com>
>>>
>>> wrote:
>>> > The clients we use have memory ranging from 4GB to 8GB. OOM was
>>> produced on all these clients....some sooner, some little later, bit
>>> always was seen.>
>>> >
>>> > The workaround is still stable for more than 48 hours now.>
>>> >
>>> >
>>>
>>
Re: Ignite 2.7.0: Ignite client:: memory leak
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
We did not change it since it should never be a problem anymore. The
rationale is that I can't prove that some history size value will work and
cause no further issues.
Regards,
--
Ilya Kasnacheev
вт, 6 авг. 2019 г. в 22:25, Denis Magda <dm...@apache.org>:
> Andrey, Ilya, as part of IGNITE-11767, have we set the history size to 0
> for the clients? If haven't what is a rationale?
>
> -
> Denis
>
>
> On Tue, Aug 6, 2019 at 5:56 AM Andrei Aleksandrov <ae...@gmail.com>
> wrote:
>
>> Hi Mahesh,
>>
>> Yes, it's a problem related to IGNITE_EXCHANGE_HISTORY_SIZE. Ignite
>> stored the data for the last 1000 exchanges.
>>
>> It generally can be required for the case when the coordinator was
>> changed and new coordinator required to load last exchange history.
>>
>> Exist two problems here:
>>
>> 1)Client nodes can't be a coordinator. So there is no reason to store
>> 1000 entries there. Will be better to set this option to some small
>> value or zero for client nodes.
>> 2)Server nodes also don't require 1000 entries. The required number of
>> exchange history can depend on the number of server nodes. I suggest
>> change the default value to small value.
>>
>> Here is the ticket related to this problem:
>>
>> https://issues.apache.org/jira/browse/IGNITE-11767
>>
>> It fixed and should be available in Ignite 2.8 where these exchanges
>> will take less memory.
>>
>> BR,
>> Andrei
>>
>> On 2019/08/03 01:09:19, Mahesh Renduchintala <m....@aline-consulting.com>
>> wrote:
>> > The clients we use have memory ranging from 4GB to 8GB. OOM was
>> produced on all these clients....some sooner, some little later, bit
>> always was seen.>
>> >
>> > The workaround is still stable for more than 48 hours now.>
>> >
>> >
>>
>
Re: Ignite 2.7.0: Ignite client:: memory leak
Posted by Denis Magda <dm...@apache.org>.
Andrey, Ilya, as part of IGNITE-11767, have we set the history size to 0
for the clients? If haven't what is a rationale?
-
Denis
On Tue, Aug 6, 2019 at 5:56 AM Andrei Aleksandrov <ae...@gmail.com>
wrote:
> Hi Mahesh,
>
> Yes, it's a problem related to IGNITE_EXCHANGE_HISTORY_SIZE. Ignite
> stored the data for the last 1000 exchanges.
>
> It generally can be required for the case when the coordinator was
> changed and new coordinator required to load last exchange history.
>
> Exist two problems here:
>
> 1)Client nodes can't be a coordinator. So there is no reason to store
> 1000 entries there. Will be better to set this option to some small
> value or zero for client nodes.
> 2)Server nodes also don't require 1000 entries. The required number of
> exchange history can depend on the number of server nodes. I suggest
> change the default value to small value.
>
> Here is the ticket related to this problem:
>
> https://issues.apache.org/jira/browse/IGNITE-11767
>
> It fixed and should be available in Ignite 2.8 where these exchanges
> will take less memory.
>
> BR,
> Andrei
>
> On 2019/08/03 01:09:19, Mahesh Renduchintala <m....@aline-consulting.com>
> wrote:
> > The clients we use have memory ranging from 4GB to 8GB. OOM was
> produced on all these clients....some sooner, some little later, bit
> always was seen.>
> >
> > The workaround is still stable for more than 48 hours now.>
> >
> >
>
Re: Ignite 2.7.0: Ignite client:: memory leak
Posted by Andrei Aleksandrov <ae...@gmail.com>.
Hi Mahesh,
Yes, it's a problem related to IGNITE_EXCHANGE_HISTORY_SIZE. Ignite
stored the data for the last 1000 exchanges.
It generally can be required for the case when the coordinator was
changed and new coordinator required to load last exchange history.
Exist two problems here:
1)Client nodes can't be a coordinator. So there is no reason to store
1000 entries there. Will be better to set this option to some small
value or zero for client nodes.
2)Server nodes also don't require 1000 entries. The required number of
exchange history can depend on the number of server nodes. I suggest
change the default value to small value.
Here is the ticket related to this problem:
https://issues.apache.org/jira/browse/IGNITE-11767
It fixed and should be available in Ignite 2.8 where these exchanges
will take less memory.
BR,
Andrei
On 2019/08/03 01:09:19, Mahesh Renduchintala <m....@aline-consulting.com>
wrote:
> The clients we use have memory ranging from 4GB to 8GB. OOM was
produced on all these clients....some sooner, some little later, bit
always was seen.>
>
> The workaround is still stable for more than 48 hours now.>
>
>
Re: Ignite 2.7.0: Ignite client:: memory leak
Posted by Mahesh Renduchintala <ma...@aline-consulting.com>.
The clients we use have memory ranging from 4GB to 8GB. OOM was produced on all these clients....some sooner, some little later, bit always was seen.
The workaround is still stable for more than 48 hours now.
Re: Ignite 2.7.0: Ignite client:: memory leak
Posted by Denis Magda <dm...@apache.org>.
Mahesh,
Hmm, yes, I remember one use case when the OOM was addressed by tweaking
IGNITE_EXCHANGE_HISTORY_SIZE. In that scenario, the environment had not
much RAM and the team couldn't allocate enough RAM for Java Heap.
How about your scenario? Have you tried to give 2GB or more to the app?
-
Denis
On Thu, Aug 1, 2019 at 8:16 PM Mahesh Renduchintala <
mahesh.renduchintala@aline-consulting.com> wrote:
> Denis,
> Thanks. Meanwhile, we made some progress...the via workaround seems to use
> these two flags in the thick client config
> -DIGNITE_SKIP_PARTITION_SIZE_VALIDATION -DIGNITE_EXCHANGE_HISTORY_SIZE=8
> So far, we haven't seen clients going OOM for about 24 hours (still
> watching)
> Based on the logs, you could see there is a partition map exchanged among
> the clients and feels like messages continuously get stored on the clients
> causing this problem.
>
>