You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Denis Magda <dm...@gridgain.com> on 2019/04/20 01:34:38 UTC
Re: Ignite DataStreamer Memory Problems
Looping in the dev list.
Community does it remind you any memory leak addressed in the master? What
do we need to get down to the issue.
Denis
On Friday, April 19, 2019, kellan <ke...@gmail.com> wrote:
> After doing additional tests to isolate the issue, it looks like Ignite is
> having a problem releasing Internal memory of cache objects passed into the
> NIO ByteBuffers that back the DataStreamer objects. At first I thought this
> might be on account of my Avro's ByteBuffers that get transformed into byte
> arrays before being loaded into the Ignite DataStreamers, but I can run my
> application without the DataStreamers (otherwise exactly the same) and
> there
> is not memory leak.
>
> I've posted more about it on StackOverflow:
> https://stackoverflow.com/questions/55752357/possible-
> memory-leak-in-ignite-datastreamer
>
> I'm trying to productionalize an Ignite Cluster in Kubernetes and can't
> move
> forward until I can solve this problem. Is there anyone who's used
> DataStreamers to do heavy write loads in a k8s environment who has any
> insight into what would be causing this?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
--
Denis Magda
Re: Ignite DataStreamer Memory Problems
Posted by kellan <ke...@gmail.com>.
At this point I've spent enough time on this problem and can move on with my
project without using @QueryTextField--I'm just letting anyone who's
concerned know what I've seen in case you want to probe into this issue any
further.
I've taken the time to write a reproducer that can be easily run on any
machine, go ahead and run it based on my instructions and you can see
whatever logs you'd like to see for yourself. It runs with 4GB of heap
default, not 1, though feel free to adjust it. With 10GB of durable memory
and 4GB of Heap and a 22GB memory limit on the container, it will consume
memory up until the limit triggering an OOM kill in Docker.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite DataStreamer Memory Problems
Posted by Evgenii Zhuravlev <e....@gmail.com>.
Hi,
Lucene indexes are stored in the heap, while I see that in reproducer
you've limited heap size to 1gb. Are you sure that you used these JVM opts?
Can you please share logs from your run, so I can check the heap usage?
Best Regards,
Evgenii
вт, 30 апр. 2019 г. в 00:23, kellan <ke...@gmail.com>:
> The issue seems to be with the @QueryTextField annotation. Unless Lucene
> indexes are supposed to be eating up all this memory, in which case it
> might
> be worth improving your documentation.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Ignite DataStreamer Memory Problems
Posted by kellan <ke...@gmail.com>.
The issue seems to be with the @QueryTextField annotation. Unless Lucene
indexes are supposed to be eating up all this memory, in which case it might
be worth improving your documentation.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite DataStreamer Memory Problems
Posted by kellan <ke...@gmail.com>.
Here is a reproducible example of the DataStreamer memory leak:
https://github.com/kellanburket/ignite-leak
I've also added a public image to DockerHub: miraco/ignite:leak
This can be run on a machine with at least 22GB of memory available to
Docker and probably 50GB of storage between WAL and persistent storage, just
to be safe.
I'm following the guidelines here:
https://apacheignite.readme.io/docs/durable-memory-tuning#section-share-ram
10GB of Durable Memory
4GB of Heap
with a 22GB memory limit in Docker that adds up to about 63% of overall RAM
Now run this container: (You adjust the cpus as needed. I'm using AWS r4.4xl
nodes with 16 cores running Amazon Linux):
docker run -v $LOCAL_STORAGE:$CONTAINER_STORAGE -v $LOCAL_WAL:$CONTAINER_WAL
-m 22G --cpus=12 --memory-swappiness 0 --name ignite.leak -d
miraco/ignite:leak
I would expect memory usage to stabilize somewhere around 18-19GB (4GB Heap
+ 10GB Durable + 640M WAL + 2GB Checkpoint Buffer + 1-2GB jdk overhead), but
instead usage per docker stats rises to the container limit forcing an OOM
kill. Feel free to increase the memory limit above 22GB. Results should be
the same though it make take longer to get there.
Now this is interesting. If I replace the cache value type, which is
Array[Byte] with a Long and run it again, memory usage eventually stabilizes
at around 19-20GB:
docker run -v $LOCAL_STORAGE:$CONTAINER_STORAGE -v $LOCAL_WAL:$CONTAINER_WAL
-e VALUE_TYPE=ValueLong -m 22G --cpus=12 --memory-swappiness 0 --name
ignite.leak -d miraco/ignite:leak
Is there something I'm missing here, or is this a bug?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite DataStreamer Memory Problems
Posted by kellan <ke...@gmail.com>.
Ignite Version: 2.7.0
Ignite Config:
https://gist.github.com/kellanburket/73971d076a9b2d4f001b073d02e2343a
Java Process: /opt/jdk/bin/java -XX:+AggressiveOpts
-XX:NativeMemoryTracking=detail -Xms24G -Xmx24G
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dumps/oom.bin
-XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC
-XX:MaxDirectMemorySize=256M -Duser.timezone=GMT
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.port=49112
-Dcom.sun.management.jmxremote.rmi.port=49112
-Djava.rmi.server.hostname=127.0.0.1 -DIGNITE_WAL_MMAP=true
-Djdk.nio.maxCachedBufferSize=262144 -DIGNITE_QUIET=true
-DIGNITE_SUCCESS_FILE=/opt/ignite/apache-ignite-2.7.0-bin/work/ignite_success_77a36388-73e4-4de6-9988-27e62775c3fc
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=49112
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-DIGNITE_HOME=/opt/ignite/apache-ignite-2.7.0-bin
-DIGNITE_PROG_NAME=/opt/ignite/apache-ignite-2.7.0-bin/bin/ignite.sh -cp
/opt/ignite/apache-ignite-2.7.0-bin/libs/*:/opt/ignite/apache-ignite-2.7.0-bin/libs/ignite-indexing/*:/opt/ignite/apache-ignite-2.7.0-bin/libs/ignite-kubernetes/*:/opt/ignite/apache-ignite-2.7.0-bin/libs/ignite-spark/*:/opt/ignite/apache-ignite-2.7.0-bin/libs/ignite-spring/*:/opt/ignite/apache-ignite-2.7.0-bin/libs/ignite-zookeeper/*:/opt/ignite/apache-ignite-2.7.0-bin/libs/licenses/*
org.apache.ignite.startup.cmdline.CommandLineStartup
/opt/ignite/apache-ignite-2.7.0-bin/config/default-config.xml
I've already tried running with walMode=None, but I'll try it again just to
confirm
I'll put together a shareable reproducer today.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite DataStreamer Memory Problems
Posted by Stanislav Lukyanov <st...@gmail.com>.
Can you share your full configuration (Ignite config and JVM options) and
the server logs of Ignite?
Which version of Ignite you use?
Can you confirm that on this version and configuration simply disabling
Ignite persistence removes the problem?
If yes, can you try running with walMode=NONE? It will help to rule out at
least some possibilities.
Also, if you can share a reproducer to this problem it should be easy for
us to debug this.
Stan
On Tue, Apr 23, 2019 at 6:42 AM kellan <ke...@gmail.com> wrote:
> Any suggestions from where I can go from here? I'd like to find a way to
> isolate this problem before I have to look into another storage/grid
> solutions. A lot of work has gone into integrating Ignite into our
> platform,
> and I'd really hate to start from scratch. I can provide as much
> information
> as needed to help pinpoint this problem/do additional tests on my end.
>
> Are there any projects out there that have successfully run Ignite on
> Kubernetes with Persistence and a high-volume write load?
>
> I've been looking into using third-party persistence but we require SQL
> queries to fetch the bulk of our data and it seems like this isn't really
> possible with Cassandra, et al, unless I can know in advance what data
> needs
> to be loaded into memory. Is that a safe assumption to make?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Ignite DataStreamer Memory Problems
Posted by kellan <ke...@gmail.com>.
Any suggestions from where I can go from here? I'd like to find a way to
isolate this problem before I have to look into another storage/grid
solutions. A lot of work has gone into integrating Ignite into our platform,
and I'd really hate to start from scratch. I can provide as much information
as needed to help pinpoint this problem/do additional tests on my end.
Are there any projects out there that have successfully run Ignite on
Kubernetes with Persistence and a high-volume write load?
I've been looking into using third-party persistence but we require SQL
queries to fetch the bulk of our data and it seems like this isn't really
possible with Cassandra, et al, unless I can know in advance what data needs
to be loaded into memory. Is that a safe assumption to make?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite DataStreamer Memory Problems
Posted by kellan <ke...@gmail.com>.
No luck with the changed configuration. Memory still continues to rise until
the Kubernetes limit (110GB), then crashes. This is output I pulled from
jcmd at some point before the crash. I can post the detailed memory report
if that helps.
Total: reserved=84645150KB, committed=83359362KB
- Java Heap (reserved=25165824KB, committed=25165824KB)
(mmap: reserved=25165824KB,
committed=25165824KB)
- Class (reserved=1121992KB, committed=80356KB)
(classes #11821)
(malloc=1736KB #20912)
(mmap: reserved=1120256KB, committed=78620KB)
- Thread (reserved=198099KB, committed=198099KB)
(thread #193)
(stack: reserved=197248KB, committed=197248KB)
(malloc=626KB #975)
(arena=225KB #380)
- Code (reserved=260571KB, committed=65571KB)
(malloc=10971KB #16284)
(mmap: reserved=249600KB, committed=54600KB)
- GC (reserved=1047369KB, committed=1047369KB)
(malloc=80713KB #57810)
(mmap: reserved=966656KB, committed=966656KB)
- Compiler (reserved=597KB, committed=597KB)
(malloc=467KB #1235)
(arena=131KB #7)
- Internal (reserved=56763248KB, committed=56763248KB)
(malloc=56763216KB #1063361)
(mmap: reserved=32KB, committed=32KB)
- Symbol (reserved=17245KB, committed=17245KB)
(malloc=14680KB #138104)
(arena=2565KB #1)
- Native Memory Tracking (reserved=20852KB, committed=20852KB)
(malloc=453KB #6407)
(tracking overhead=20399KB)
- Arena Chunk (reserved=201KB, committed=201KB)
(malloc=201KB)
- Unknown (reserved=49152KB, committed=0KB)
(mmap: reserved=49152KB, committed=0KB)
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite DataStreamer Memory Problems
Posted by Stanislav Lukyanov <st...@gmail.com>.
I've put a full answer on SO -
https://stackoverflow.com/questions/55752357/possible-memory-leak-in-ignite-datastreamer/55786023#55786023
.
In short, so far it doesn't look like a memory leak to me - just a
misconfiguration.
There is a memory pool in JVM for direct memory buffers which is by default
bounded by the value of `-Xmx`. Most applications would use minuscule
amount of it, but in some it can grow - and grow to the size of the heap,
making your total Java usage not roughly `heap + data region` but `heap *
2 + data region`.
Set walSegmentSize=64mb and -XX:MaxDirectMemorySize=256mb and I think it's
going to be OK.
Stan
On Sun, Apr 21, 2019 at 11:51 AM Denis Magda <dm...@apache.org> wrote:
> Hello,
>
> Copying Evgeniy and Stan, our community experts who'd guide you through.
> In the meantime, please try to capture the OOM with this approach:
>
> https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html
>
> -
> Denis
>
>
> On Sun, Apr 21, 2019 at 8:49 AM kellan <ke...@gmail.com> wrote:
>
>> Update: I've been able to confirm a couple more details:
>>
>> 1. I'm experiencing the same leak with put, putAll as I am with the
>> DataStreamer
>> 2. The problem is resolved when persistence is turned off
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
Re: Ignite DataStreamer Memory Problems
Posted by Denis Magda <dm...@apache.org>.
Hello,
Copying Evgeniy and Stan, our community experts who'd guide you through. In
the meantime, please try to capture the OOM with this approach:
https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html
-
Denis
On Sun, Apr 21, 2019 at 8:49 AM kellan <ke...@gmail.com> wrote:
> Update: I've been able to confirm a couple more details:
>
> 1. I'm experiencing the same leak with put, putAll as I am with the
> DataStreamer
> 2. The problem is resolved when persistence is turned off
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Ignite DataStreamer Memory Problems
Posted by kellan <ke...@gmail.com>.
Update: I've been able to confirm a couple more details:
1. I'm experiencing the same leak with put, putAll as I am with the
DataStreamer
2. The problem is resolved when persistence is turned off
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/