You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Jason <fq...@outlook.com> on 2016/07/29 17:03:25 UTC
How is memory used for the Ignite server node?
hi Ignite team,
I've a 21 server nodes Ignite cluster and found that their memory usage has
a very big difference:
the min is ~28G, and the max is ~45G.
our total data capacity is ~90G (key is tens of bytes and value is ~2-3k),
with 2 backups and few columns for sql query and based on the capacity
planning formula in your doc:
memory for each node: 90G * 3 * 1.3 / 21 ~= 17G
And now, we've set the JVM max memory to 10G and use off_heap with max size
30G.
Checked the JMX for the min memory and max memory nodes, found that:
for max memory node (45G),
total entries: ~6.2M, allocated off-heap size: ~17.3G, heap: 9G
for min memory node (28G),
total entries: ~5.5M, allocated off-heap size: ~14.5G, heap: 7G
So I wonder where other memory is used (max memory node: 45G - 17.3G - 9G ~=
17G, even for min memory node: 28G - 14.5G - 7G ~= 6.5G) ?
BTW, we use off-heap, why does it still need so many heap (9G versus 17.3G,
7G versus 14.5G)? If I reduce the max JVM memory, will it throw OOM
exception? If not, any impact to the latency?
would you like to share some experience data during production?
Thanks,
-Jason
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-is-memory-used-for-the-Ignite-server-node-tp6626.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: How is memory used for the Ignite server node?
Posted by vkulichenko <va...@gmail.com>.
Hi Jason,
Provided capacity planning guide can give you rough estimate to start with
and you will have to test case by case after that. I actually think it's way
more risky if you don't do that.
As for the memory consumed by process, remember that both JVM and CLR are
managing their own memory, so it can actually be consumed by temporary
object that are not GCed yet, for example. It also depends on what kind of
load you create on the cluster (only cache, compute, etc.). Generally I
recommend at least 4GB heap memory in case the data is off-heap. If you
don't experience OOM errors or GC pauses, you're good.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-is-memory-used-for-the-Ignite-server-node-tp6626p6899.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: How is memory used for the Ignite server node?
Posted by Jason <fq...@outlook.com>.
hi Val,
The OffheapAllocatedSize is the real allocated memory for the Java Unsafe,
right?
And the heap memory is for the JVM managed memory, right?
Then where's the gap memory between (heap memory + OffheapAllocatedSize
memory) and the real memory of the Apache.Ignite.exe for .net version? Is it
used by the .net CLR? Just because .net CLR and JVM are in the same one
process, is there any metrics in Ignite to measure this?
Thanks,
-Jason
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-is-memory-used-for-the-Ignite-server-node-tp6626p6645.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: How is memory used for the Ignite server node?
Posted by Jason <fq...@outlook.com>.
Thanks Val.
>> Heap memory is not managed by Ignite, so such fluctuations are always
>> possible. You can try to decrease heap size to 4g, for example, and check
>> how it works. Depending on your use case this can be enough or not (but I
>> would not recommend less).
what does capacity planning
<https://apacheignite.readme.io/docs/capacity-planning> mean? it doesn't
include all the memory that is needed? for big off_heap, it still needs
extra heap memory to process? then how to estimate the heap memory usage?
would you like to share some production experience data?
>> As for off-heap, this can depend on different factors. Are your entries
>> typically of the same size? Do you use any kind of collocation? Ignite
>> affinity tries to evenly distribute entries across nodes, and if you
>> simple put a lot of entries of the same size, consumed memory should be
>> the same on all nodes. But in real use cases that's not always true.
The entries are in the size of ~2k-3k. no collocation, only one entry type.
It's OK for all the nodes not to have the same entries, but my concern is
the memory which is much bigger than that of capacity planning
<https://apacheignite.readme.io/docs/capacity-planning> .
If there isn't a good guide for the capacity planning, and it can only be
tested case by case, it should be very risky for the cache's reliability and
cost with big data.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-is-memory-used-for-the-Ignite-server-node-tp6626p6644.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: How is memory used for the Ignite server node?
Posted by vkulichenko <va...@gmail.com>.
Hi Jason,
Heap memory is not managed by Ignite, so such fluctuations are always
possible. You can try to decrease heap size to 4g, for example, and check
how it works. Depending on your use case this can be enough or not (but I
would not recommend less).
As for off-heap, this can depend on different factors. Are your entries
typically of the same size? Do you use any kind of collocation? Ignite
affinity tries to evenly distribute entries across nodes, and if you simple
put a lot of entries of the same size, consumed memory should be the same on
all nodes. But in real use cases that's not always true.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-is-memory-used-for-the-Ignite-server-node-tp6626p6632.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.