You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Ravi Makwana <ra...@hotelhub.com> on 2020/10/24 11:03:26 UTC
Ignite Client Node Stopped With Out Of Memory
Hi,
We are using Apache Ignite 2.7.0 binary and servers are using Linux OS &
app servers are using Windows OS.We are using Apache Ignite .Net APIs.
Recently we have noticed that our application is stopping due to a client
node throwing Out Of Memory error which we have seen in ignite
client node log file.
App server has 32 GB RAM & We are specifying JVM Heap = 8 GB & Data Region
Min = 1 GB & Max = 10 GB.
*Exception:*
class org.apache.ignite.IgniteException: Platform
error:System.OutOfMemoryException: Insufficient memory to continue the
execution of the program.
at System.Runtime.InteropServices.Marshal.ReAllocHGlobal(IntPtr pv,
IntPtr cb)
at
Apache.Ignite.Core.Impl.Memory.PlatformMemoryUtils.ReallocatePooled(Int64
memPtr, Int32 cap)
at
Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.MemoryReallocate(Int64
memPtr, Int64 cap, Int64 unused, Void* arg)
at
Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.InLongLongLongObjectOutLong(Int32
type, Int64 val1, Int64 val2, Int64 val3, IntPtr arg)
at
org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.loggerLog(PlatformProcessorImpl.java:404)
at
org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutLong(PlatformProcessorImpl.java:460)
at
org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutLong(PlatformProcessorImpl.java:512)
at
org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67)
at
org.apache.ignite.internal.processors.platform.callback.PlatformCallbackUtils.inLongLongLongObjectOutLong(Native
Method)
at
org.apache.ignite.internal.processors.platform.callback.PlatformCallbackGateway.memoryReallocate(PlatformCallbackGateway.java:840)
at
org.apache.ignite.internal.processors.platform.memory.PlatformExternalMemory.reallocate(PlatformExternalMemory.java:48)
at
org.apache.ignite.internal.processors.platform.memory.PlatformOutputStreamImpl.ensureCapacity(PlatformOutputStreamImpl.java:305)
at
org.apache.ignite.internal.processors.platform.memory.PlatformOutputStreamImpl.unsafeEnsure(PlatformOutputStreamImpl.java:222)
at
org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteBinaryObject(BinaryWriterExImpl.java:962)
at
org.apache.ignite.internal.binary.BinaryClassDescriptor.write(BinaryClassDescriptor.java:744)
at
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:223)
at
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:164)
at
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:151)
at
org.apache.ignite.internal.binary.BinaryWriterExImpl.writeObjectDetached(BinaryWriterExImpl.java:1506)
at
org.apache.ignite.internal.processors.platform.cache.query.PlatformQueryCursor.write(PlatformQueryCursor.java:43)
at
org.apache.ignite.internal.processors.platform.cache.query.PlatformQueryCursor.write(PlatformQueryCursor.java:28)
at
org.apache.ignite.internal.processors.platform.cache.query.PlatformAbstractQueryCursor.processOutStream(PlatformAbstractQueryCursor.java:85)
at
org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.outStream(PlatformTargetProxyImpl.java:93)
Can please suggest what is the root cause for the above issue?
Thanks & Regards,
Re: Ignite Client Node Stopped With Out Of Memory
Posted by Pavel Tupitsyn <pt...@apache.org>.
On of these, depending on your query type:
* new ScanQuery<K, V>() { PageSize = 5 }
* new SqlQuery() { PageSize = 5 }
On Wed, Oct 28, 2020 at 5:24 PM Ravi Makwana <ra...@hotelhub.com>
wrote:
> Hi Paval,
>
> As we are not setting explicitly QueryBase.Pagesize for SqlQuery and
> SqlFieldQuery so default will be used as 1024.
>
> We have not found so far any example by looking the same we can try to
> explicitly set the Query base.Pagesize.
>
> Can we have any reference by checking that we can try to set the lower
> value and can able to replicate the scenario?
>
>
> Thanks,
>
> On Wed, 28 Oct, 2020, 7:53 pm Ravi Makwana, <ra...@hotelhub.com>
> wrote:
>
>>
>>
>>
>>
>> On Wed, 28 Oct, 2020, 5:55 pm Pavel Tupitsyn, <pt...@apache.org>
>> wrote:
>>
>>> I found a bug in Ignite [1] which probably causes the issue on your side.
>>>
>>> Looks like you are running a query (is it a ScanQuery or SqlQuery?) and
>>> the size of one results page exceeds 2GB.
>>> Please try using a smaller value for *QueryBase.PageSize*.
>>>
>>> If you use the default value of 1024, your cache entries are very large.
>>> What does the data look like?
>>>
>>> [1] https://issues.apache.org/jira/browse/IGNITE-13635
>>>
>>> On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana <ra...@hotelhub.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Our service is running with 64 bit and we have verified the same in our
>>>> app server too.
>>>>
>>>> Any finding from the logs?
>>>>
>>>> Is there any way to replicate it?
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn <pt...@apache.org>
>>>> wrote:
>>>>
>>>>> Looks like the app is running in 32 bit mode, which can't use more
>>>>> than 2GB of memory.
>>>>> JVM and memory regions pre-allocate all of it, leaving nothing for
>>>>> .NET to use.
>>>>>
>>>>> Please check the `Platform` column in the Task Manager - does it say
>>>>> `32 bit`?
>>>>> If yes, then try disabling `Prefer 32 bit` in the project properties.
>>>>>
>>>>> On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana <
>>>>> ravi.makwana@hotelhub.com> wrote:
>>>>>
>>>>>> Hi Pavel,
>>>>>>
>>>>>> Thanks for the information.
>>>>>>
>>>>>> We have noticed our service is taking max 2GB memory which is noticed
>>>>>> in task manager and stopping app pool at same time server memory
>>>>>> utilization is 80% so we have still memory spare 20%.
>>>>>>
>>>>>> Still we are not able to correlate the issue, I would like to share
>>>>>> some more logs. Could you suggest what is happening at the moment?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>
Re: Ignite Client Node Stopped With Out Of Memory
Posted by Ravi Makwana <ra...@hotelhub.com>.
Hi Paval,
As we are not setting explicitly QueryBase.Pagesize for SqlQuery and
SqlFieldQuery so default will be used as 1024.
We have not found so far any example by looking the same we can try to
explicitly set the Query base.Pagesize.
Can we have any reference by checking that we can try to set the lower
value and can able to replicate the scenario?
Thanks,
On Wed, 28 Oct, 2020, 7:53 pm Ravi Makwana, <ra...@hotelhub.com>
wrote:
>
>
>
>
> On Wed, 28 Oct, 2020, 5:55 pm Pavel Tupitsyn, <pt...@apache.org>
> wrote:
>
>> I found a bug in Ignite [1] which probably causes the issue on your side.
>>
>> Looks like you are running a query (is it a ScanQuery or SqlQuery?) and
>> the size of one results page exceeds 2GB.
>> Please try using a smaller value for *QueryBase.PageSize*.
>>
>> If you use the default value of 1024, your cache entries are very large.
>> What does the data look like?
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-13635
>>
>> On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana <ra...@hotelhub.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Our service is running with 64 bit and we have verified the same in our
>>> app server too.
>>>
>>> Any finding from the logs?
>>>
>>> Is there any way to replicate it?
>>>
>>> Thanks,
>>>
>>>
>>> On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn <pt...@apache.org>
>>> wrote:
>>>
>>>> Looks like the app is running in 32 bit mode, which can't use more than
>>>> 2GB of memory.
>>>> JVM and memory regions pre-allocate all of it, leaving nothing for .NET
>>>> to use.
>>>>
>>>> Please check the `Platform` column in the Task Manager - does it say
>>>> `32 bit`?
>>>> If yes, then try disabling `Prefer 32 bit` in the project properties.
>>>>
>>>> On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana <ra...@hotelhub.com>
>>>> wrote:
>>>>
>>>>> Hi Pavel,
>>>>>
>>>>> Thanks for the information.
>>>>>
>>>>> We have noticed our service is taking max 2GB memory which is noticed
>>>>> in task manager and stopping app pool at same time server memory
>>>>> utilization is 80% so we have still memory spare 20%.
>>>>>
>>>>> Still we are not able to correlate the issue, I would like to share
>>>>> some more logs. Could you suggest what is happening at the moment?
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
Re: Ignite Client Node Stopped With Out Of Memory
Posted by Ravi Makwana <ra...@hotelhub.com>.
On Wed, 28 Oct, 2020, 5:55 pm Pavel Tupitsyn, <pt...@apache.org> wrote:
> I found a bug in Ignite [1] which probably causes the issue on your side.
>
> Looks like you are running a query (is it a ScanQuery or SqlQuery?) and
> the size of one results page exceeds 2GB.
> Please try using a smaller value for *QueryBase.PageSize*.
>
> If you use the default value of 1024, your cache entries are very large.
> What does the data look like?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-13635
>
> On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana <ra...@hotelhub.com>
> wrote:
>
>> Hi,
>>
>> Our service is running with 64 bit and we have verified the same in our
>> app server too.
>>
>> Any finding from the logs?
>>
>> Is there any way to replicate it?
>>
>> Thanks,
>>
>>
>> On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn <pt...@apache.org>
>> wrote:
>>
>>> Looks like the app is running in 32 bit mode, which can't use more than
>>> 2GB of memory.
>>> JVM and memory regions pre-allocate all of it, leaving nothing for .NET
>>> to use.
>>>
>>> Please check the `Platform` column in the Task Manager - does it say `32
>>> bit`?
>>> If yes, then try disabling `Prefer 32 bit` in the project properties.
>>>
>>> On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana <ra...@hotelhub.com>
>>> wrote:
>>>
>>>> Hi Pavel,
>>>>
>>>> Thanks for the information.
>>>>
>>>> We have noticed our service is taking max 2GB memory which is noticed
>>>> in task manager and stopping app pool at same time server memory
>>>> utilization is 80% so we have still memory spare 20%.
>>>>
>>>> Still we are not able to correlate the issue, I would like to share
>>>> some more logs. Could you suggest what is happening at the moment?
>>>>
>>>> Thanks,
>>>>
>>>>
Re: Ignite Client Node Stopped With Out Of Memory
Posted by Pavel Tupitsyn <pt...@apache.org>.
I found a bug in Ignite [1] which probably causes the issue on your side.
Looks like you are running a query (is it a ScanQuery or SqlQuery?) and the
size of one results page exceeds 2GB.
Please try using a smaller value for *QueryBase.PageSize*.
If you use the default value of 1024, your cache entries are very large.
What does the data look like?
[1] https://issues.apache.org/jira/browse/IGNITE-13635
On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana <ra...@hotelhub.com>
wrote:
> Hi,
>
> Our service is running with 64 bit and we have verified the same in our
> app server too.
>
> Any finding from the logs?
>
> Is there any way to replicate it?
>
> Thanks,
>
>
> On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn <pt...@apache.org> wrote:
>
>> Looks like the app is running in 32 bit mode, which can't use more than
>> 2GB of memory.
>> JVM and memory regions pre-allocate all of it, leaving nothing for .NET
>> to use.
>>
>> Please check the `Platform` column in the Task Manager - does it say `32
>> bit`?
>> If yes, then try disabling `Prefer 32 bit` in the project properties.
>>
>> On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana <ra...@hotelhub.com>
>> wrote:
>>
>>> Hi Pavel,
>>>
>>> Thanks for the information.
>>>
>>> We have noticed our service is taking max 2GB memory which is noticed in
>>> task manager and stopping app pool at same time server memory utilization
>>> is 80% so we have still memory spare 20%.
>>>
>>> Still we are not able to correlate the issue, I would like to share some
>>> more logs. Could you suggest what is happening at the moment?
>>>
>>> Thanks,
>>>
>>>
Re: Ignite Client Node Stopped With Out Of Memory
Posted by Ravi Makwana <ra...@hotelhub.com>.
Hi,
Our service is running with 64 bit and we have verified the same in our app
server too.
Any finding from the logs?
Is there any way to replicate it?
Thanks,
On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn <pt...@apache.org> wrote:
> Looks like the app is running in 32 bit mode, which can't use more than
> 2GB of memory.
> JVM and memory regions pre-allocate all of it, leaving nothing for .NET to
> use.
>
> Please check the `Platform` column in the Task Manager - does it say `32
> bit`?
> If yes, then try disabling `Prefer 32 bit` in the project properties.
>
> On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana <ra...@hotelhub.com>
> wrote:
>
>> Hi Pavel,
>>
>> Thanks for the information.
>>
>> We have noticed our service is taking max 2GB memory which is noticed in
>> task manager and stopping app pool at same time server memory utilization
>> is 80% so we have still memory spare 20%.
>>
>> Still we are not able to correlate the issue, I would like to share some
>> more logs. Could you suggest what is happening at the moment?
>>
>> Thanks,
>>
>>
Re: Ignite Client Node Stopped With Out Of Memory
Posted by Pavel Tupitsyn <pt...@apache.org>.
Looks like the app is running in 32 bit mode, which can't use more than 2GB
of memory.
JVM and memory regions pre-allocate all of it, leaving nothing for .NET to
use.
Please check the `Platform` column in the Task Manager - does it say `32
bit`?
If yes, then try disabling `Prefer 32 bit` in the project properties.
On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana <ra...@hotelhub.com>
wrote:
> Hi Pavel,
>
> Thanks for the information.
>
> We have noticed our service is taking max 2GB memory which is noticed in
> task manager and stopping app pool at same time server memory utilization
> is 80% so we have still memory spare 20%.
>
> Still we are not able to correlate the issue, I would like to share some
> more logs. Could you suggest what is happening at the moment?
>
> Thanks,
>
>
Re: Ignite Client Node Stopped With Out Of Memory
Posted by Ravi Makwana <ra...@hotelhub.com>.
Hi Pavel,
Thanks for the information.
We have noticed our service is taking max 2GB memory which is noticed in
task manager and stopping app pool at same time server memory utilization
is 80% so we have still memory spare 20%.
Still we are not able to correlate the issue, I would like to share some
more logs. Could you suggest what is happening at the moment?
Thanks,
Re: Ignite Client Node Stopped With Out Of Memory
Posted by Pavel Tupitsyn <pt...@apache.org>.
Hi Ravi,
The exception indicates that Ignite.NET failed to allocate unmanaged memory
on .NET side while trying to pass query data from Java to .NET.
This indicates that your system has run out of memory. Possible reasons are:
* Memory is consumed by other apps
* Memory is consumed by this app
* Memory leak in your code
* Memory leak in Ignite.NET code
I would suggest the following steps:
* Find out which app consumes the memory. Are we sure it is the app with
Ignite?
* If yes, run a memory profiler of your choice to figure out what consumes
the memory
Thanks,
Pavel
On Sat, Oct 24, 2020 at 2:04 PM Ravi Makwana <ra...@hotelhub.com>
wrote:
> Hi,
>
> We are using Apache Ignite 2.7.0 binary and servers are using Linux OS &
> app servers are using Windows OS.We are using Apache Ignite .Net APIs.
>
> Recently we have noticed that our application is stopping due to a client
> node throwing Out Of Memory error which we have seen in ignite
> client node log file.
>
> App server has 32 GB RAM & We are specifying JVM Heap = 8 GB & Data Region
> Min = 1 GB & Max = 10 GB.
>
> *Exception:*
>
> class org.apache.ignite.IgniteException: Platform
> error:System.OutOfMemoryException: Insufficient memory to continue the
> execution of the program.
> at System.Runtime.InteropServices.Marshal.ReAllocHGlobal(IntPtr pv,
> IntPtr cb)
> at
> Apache.Ignite.Core.Impl.Memory.PlatformMemoryUtils.ReallocatePooled(Int64
> memPtr, Int32 cap)
> at
> Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.MemoryReallocate(Int64
> memPtr, Int64 cap, Int64 unused, Void* arg)
> at
> Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.InLongLongLongObjectOutLong(Int32
> type, Int64 val1, Int64 val2, Int64 val3, IntPtr arg)
> at
> org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.loggerLog(PlatformProcessorImpl.java:404)
> at
> org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutLong(PlatformProcessorImpl.java:460)
> at
> org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutLong(PlatformProcessorImpl.java:512)
> at
> org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67)
> at
> org.apache.ignite.internal.processors.platform.callback.PlatformCallbackUtils.inLongLongLongObjectOutLong(Native
> Method)
> at
> org.apache.ignite.internal.processors.platform.callback.PlatformCallbackGateway.memoryReallocate(PlatformCallbackGateway.java:840)
> at
> org.apache.ignite.internal.processors.platform.memory.PlatformExternalMemory.reallocate(PlatformExternalMemory.java:48)
> at
> org.apache.ignite.internal.processors.platform.memory.PlatformOutputStreamImpl.ensureCapacity(PlatformOutputStreamImpl.java:305)
> at
> org.apache.ignite.internal.processors.platform.memory.PlatformOutputStreamImpl.unsafeEnsure(PlatformOutputStreamImpl.java:222)
> at
> org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteBinaryObject(BinaryWriterExImpl.java:962)
> at
> org.apache.ignite.internal.binary.BinaryClassDescriptor.write(BinaryClassDescriptor.java:744)
> at
> org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:223)
> at
> org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:164)
> at
> org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:151)
> at
> org.apache.ignite.internal.binary.BinaryWriterExImpl.writeObjectDetached(BinaryWriterExImpl.java:1506)
> at
> org.apache.ignite.internal.processors.platform.cache.query.PlatformQueryCursor.write(PlatformQueryCursor.java:43)
> at
> org.apache.ignite.internal.processors.platform.cache.query.PlatformQueryCursor.write(PlatformQueryCursor.java:28)
> at
> org.apache.ignite.internal.processors.platform.cache.query.PlatformAbstractQueryCursor.processOutStream(PlatformAbstractQueryCursor.java:85)
> at
> org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.outStream(PlatformTargetProxyImpl.java:93)
>
> Can please suggest what is the root cause for the above issue?
>
> Thanks & Regards,
>
>