You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Peter Schmitt <pe...@gmail.com> on 2016/06/28 11:32:42 UTC
Non-cluster mode
Hello Ignite-Community!
As a part of an evaluation of Ignite 1.6, I'm trying to use Ignite in a
cluster *without* cluster-mode.
It should be used as local cache (independent of the other nodes).
Therefore, the (Ignite-)nodes shouldn't try to discover other
(Ignite-)nodes.
I tried
TcpDiscoverySpi localNodeDiscovery = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Collections.singletonList("127.0.0.1"));
localNodeDiscovery.setIpFinder(ipFinder);
IgniteConfiguration configuration = new IgniteConfiguration()
.setDiscoverySpi(localNodeDiscovery)
.setDaemon(false);
Ignite ignite = Ignition.getOrStart(configuration);
to limit cluster-discovery to the local node.
In that case Ignite does nothing at all (the startup hangs).
I've read the docs about the different cluster-modes, but I couldn't find
the information I'm looking for (which works as expected).
Any hint is appreciated,
Peter
Re: Non-cluster mode
Posted by Peter Schmitt <pe...@gmail.com>.
Thank you Denis for the hint!
Kind regards
Peter
2016-07-05 14:35 GMT+02:00 Denis Magda <dm...@gridgain.com>:
> Peter,
>
> Measure it basing under your load. Sometimes it’s enough to have Java
> heaps 8 GB in size to work under significant load with off-heap data in
> hundreds of GBs in size.
>
> —
> Denis
>
> On Jul 5, 2016, at 3:20 PM, Peter Schmitt <pe...@gmail.com>
> wrote:
>
> Hi Denis,
>
> we are trying to store a huge amount of data off-heap (more than 50 GB).
> Therefore, we need to know the heap-size which is needed by Ignite to
> handle such a huge off-heap cache (which we only need to keep the heap-size
> as low as possible due to the GC overhead).
>
> Kind regards
> Peter
>
>
>
> 2016-07-05 11:41 GMT+02:00 Denis Magda <dm...@gridgain.com>:
>
>> Hi Peter,
>>
>> Basically it depends on your use case. Sometimes it’s enough 2 GB,
>> sometimes 5 GB or 10 GB. It depends on the workload.
>> However you shouldn’t allocate too big Java heaps with size more than 20
>> GB because it can lead to long stop-the-world pauses at some time.
>>
>> —
>> Denis
>>
>> On Jul 3, 2016, at 11:29 AM, Peter Schmitt <pe...@gmail.com>
>> wrote:
>>
>> Hi Val,
>>
>> after several tests I can confirm that it works with more heap-memory.
>> However, I'm not sure how much heap-memory is needed for 50+ GB
>> off-heap-data and I can't find hints for it in the docs.
>>
>> Kind regards
>> Peter
>>
>>
>>
>> 2016-06-30 22:23 GMT+02:00 vkulichenko <va...@gmail.com>:
>>
>>> Hi Peter,
>>>
>>> It sound like you just don't give enough heap memory to a node. Heap
>>> memory
>>> is still required, even if you store all the data off heap. Can you try
>>> to
>>> your JVM at least 2GB and check if this helps?
>>>
>>> -Val
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-ignite-users.70518.x6.nabble.com/Non-cluster-mode-tp5959p6028.html
>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com
>>> <http://nabble.com/>.
>>>
>>
>>
>>
>
>
Re: Non-cluster mode
Posted by Denis Magda <dm...@gridgain.com>.
Peter,
Measure it basing under your load. Sometimes it’s enough to have Java heaps 8 GB in size to work under significant load with off-heap data in hundreds of GBs in size.
—
Denis
> On Jul 5, 2016, at 3:20 PM, Peter Schmitt <pe...@gmail.com> wrote:
>
> Hi Denis,
>
> we are trying to store a huge amount of data off-heap (more than 50 GB).
> Therefore, we need to know the heap-size which is needed by Ignite to handle such a huge off-heap cache (which we only need to keep the heap-size as low as possible due to the GC overhead).
>
> Kind regards
> Peter
>
>
>
> 2016-07-05 11:41 GMT+02:00 Denis Magda <dmagda@gridgain.com <ma...@gridgain.com>>:
> Hi Peter,
>
> Basically it depends on your use case. Sometimes it’s enough 2 GB, sometimes 5 GB or 10 GB. It depends on the workload.
> However you shouldn’t allocate too big Java heaps with size more than 20 GB because it can lead to long stop-the-world pauses at some time.
>
> —
> Denis
>
>> On Jul 3, 2016, at 11:29 AM, Peter Schmitt <peter.schmitt.jee@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi Val,
>>
>> after several tests I can confirm that it works with more heap-memory.
>> However, I'm not sure how much heap-memory is needed for 50+ GB off-heap-data and I can't find hints for it in the docs.
>>
>> Kind regards
>> Peter
>>
>>
>>
>> 2016-06-30 22:23 GMT+02:00 vkulichenko <valentin.kulichenko@gmail.com <ma...@gmail.com>>:
>> Hi Peter,
>>
>> It sound like you just don't give enough heap memory to a node. Heap memory
>> is still required, even if you store all the data off heap. Can you try to
>> your JVM at least 2GB and check if this helps?
>>
>> -Val
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Non-cluster-mode-tp5959p6028.html <http://apache-ignite-users.70518.x6.nabble.com/Non-cluster-mode-tp5959p6028.html>
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com <http://nabble.com/>.
>>
>
>
Re: Non-cluster mode
Posted by Peter Schmitt <pe...@gmail.com>.
Hi Denis,
we are trying to store a huge amount of data off-heap (more than 50 GB).
Therefore, we need to know the heap-size which is needed by Ignite to
handle such a huge off-heap cache (which we only need to keep the heap-size
as low as possible due to the GC overhead).
Kind regards
Peter
2016-07-05 11:41 GMT+02:00 Denis Magda <dm...@gridgain.com>:
> Hi Peter,
>
> Basically it depends on your use case. Sometimes it’s enough 2 GB,
> sometimes 5 GB or 10 GB. It depends on the workload.
> However you shouldn’t allocate too big Java heaps with size more than 20
> GB because it can lead to long stop-the-world pauses at some time.
>
> —
> Denis
>
> On Jul 3, 2016, at 11:29 AM, Peter Schmitt <pe...@gmail.com>
> wrote:
>
> Hi Val,
>
> after several tests I can confirm that it works with more heap-memory.
> However, I'm not sure how much heap-memory is needed for 50+ GB
> off-heap-data and I can't find hints for it in the docs.
>
> Kind regards
> Peter
>
>
>
> 2016-06-30 22:23 GMT+02:00 vkulichenko <va...@gmail.com>:
>
>> Hi Peter,
>>
>> It sound like you just don't give enough heap memory to a node. Heap
>> memory
>> is still required, even if you store all the data off heap. Can you try to
>> your JVM at least 2GB and check if this helps?
>>
>> -Val
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-ignite-users.70518.x6.nabble.com/Non-cluster-mode-tp5959p6028.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com
>> <http://nabble.com>.
>>
>
>
>
Re: Non-cluster mode
Posted by Denis Magda <dm...@gridgain.com>.
Hi Peter,
Basically it depends on your use case. Sometimes it’s enough 2 GB, sometimes 5 GB or 10 GB. It depends on the workload.
However you shouldn’t allocate too big Java heaps with size more than 20 GB because it can lead to long stop-the-world pauses at some time.
—
Denis
> On Jul 3, 2016, at 11:29 AM, Peter Schmitt <pe...@gmail.com> wrote:
>
> Hi Val,
>
> after several tests I can confirm that it works with more heap-memory.
> However, I'm not sure how much heap-memory is needed for 50+ GB off-heap-data and I can't find hints for it in the docs.
>
> Kind regards
> Peter
>
>
>
> 2016-06-30 22:23 GMT+02:00 vkulichenko <valentin.kulichenko@gmail.com <ma...@gmail.com>>:
> Hi Peter,
>
> It sound like you just don't give enough heap memory to a node. Heap memory
> is still required, even if you store all the data off heap. Can you try to
> your JVM at least 2GB and check if this helps?
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Non-cluster-mode-tp5959p6028.html <http://apache-ignite-users.70518.x6.nabble.com/Non-cluster-mode-tp5959p6028.html>
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Non-cluster mode
Posted by Peter Schmitt <pe...@gmail.com>.
Hi Val,
after several tests I can confirm that it works with more heap-memory.
However, I'm not sure how much heap-memory is needed for 50+ GB
off-heap-data and I can't find hints for it in the docs.
Kind regards
Peter
2016-06-30 22:23 GMT+02:00 vkulichenko <va...@gmail.com>:
> Hi Peter,
>
> It sound like you just don't give enough heap memory to a node. Heap memory
> is still required, even if you store all the data off heap. Can you try to
> your JVM at least 2GB and check if this helps?
>
> -Val
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Non-cluster-mode-tp5959p6028.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Non-cluster mode
Posted by vkulichenko <va...@gmail.com>.
Hi Peter,
It sound like you just don't give enough heap memory to a node. Heap memory
is still required, even if you store all the data off heap. Can you try to
your JVM at least 2GB and check if this helps?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Non-cluster-mode-tp5959p6028.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Non-cluster mode
Posted by Peter Schmitt <pe...@gmail.com>.
Hi Val,
I did further tests and it looks like OutOfMemoryError shows up in case the
VM isn't fast enough with its GC. If you add e.g. cache.get in a loop for
every key, it's possible to reproduce it. Doing System.gc() every thousand
calls takes longer, but the VM has enough time for GC and it works. Nothing
is referenced permanently, because a simple cache.get call is used. But I'm
not sure how we can handle it in a high-load scenario. From my tests I
can't see the coherency between size of Off-Heap-Memory and the required
size of On-Heap-Memory to avoid a OutOfMemoryError.
Kind regards
Peter
2016-06-29 11:00 GMT+02:00 Peter Schmitt <pe...@gmail.com>:
> Hi Val,
>
> I've pushed a demo to https://github.com/ps4os/ignite_offheap_test
> The issue is that I can't reproduce it consistently. As soon as I get the
> OutOfMemoryError, I can reproduce it. And I can reproduce it with a restart
> in between.
> The demo contains a Readme as well as some TODOs and comments which
> document the goals and observed issues.
>
> Kind regards
> Peter
>
>
> 2016-06-29 3:28 GMT+02:00 vkulichenko <va...@gmail.com>:
>
>> Peter,
>>
>> This doesn't make much sense to me. With OFFHEAP_TIERED eviction policy
>> should not change anything at all, so it sounds like misconfiguration. Can
>> you provide the whole test that I can run and investigate?
>>
>> -Val
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-ignite-users.70518.x6.nabble.com/Non-cluster-mode-tp5959p5982.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
Re: Non-cluster mode
Posted by Peter Schmitt <pe...@gmail.com>.
Hi Val,
I've pushed a demo to https://github.com/ps4os/ignite_offheap_test
The issue is that I can't reproduce it consistently. As soon as I get the
OutOfMemoryError, I can reproduce it. And I can reproduce it with a restart
in between.
The demo contains a Readme as well as some TODOs and comments which
document the goals and observed issues.
Kind regards
Peter
2016-06-29 3:28 GMT+02:00 vkulichenko <va...@gmail.com>:
> Peter,
>
> This doesn't make much sense to me. With OFFHEAP_TIERED eviction policy
> should not change anything at all, so it sounds like misconfiguration. Can
> you provide the whole test that I can run and investigate?
>
> -Val
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Non-cluster-mode-tp5959p5982.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Non-cluster mode
Posted by vkulichenko <va...@gmail.com>.
Peter,
This doesn't make much sense to me. With OFFHEAP_TIERED eviction policy
should not change anything at all, so it sounds like misconfiguration. Can
you provide the whole test that I can run and investigate?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Non-cluster-mode-tp5959p5982.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Non-cluster mode
Posted by Peter Schmitt <pe...@gmail.com>.
Hi Val,
without the eviction policy, the setup breaks due to
java.lang.OutOfMemoryError: GC overhead limit exceeded
I'm not sure why the heap grows (+ it looks like GC can't free it), in case
of the mentioned (off-heap) config.
In the end almost everything should be stored off-heap.
However, maybe it's a kind of index which needs heap-space.
In that case it would be great to know how much heap-space is safe if the
off-heap size should be ~50 GB.
Kind regards
Peter
2016-06-28 21:14 GMT+02:00 vkulichenko <va...@gmail.com>:
> Peter,
>
> The only configuration that defines whether nodes join topology or not is
> discovery SPI (the one you provided in the first message)
>
> All looks fine, expect that eviction policy will be ignored in your case.
> It's used for entries that are stored in heap memory, while your cache is
> OFFHEAP_TIERED and therefore stores everything offheap.
>
> -Val
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Non-cluster-mode-tp5959p5978.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Non-cluster mode
Posted by vkulichenko <va...@gmail.com>.
Peter,
The only configuration that defines whether nodes join topology or not is
discovery SPI (the one you provided in the first message)
All looks fine, expect that eviction policy will be ignored in your case.
It's used for entries that are stored in heap memory, while your cache is
OFFHEAP_TIERED and therefore stores everything offheap.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Non-cluster-mode-tp5959p5978.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Non-cluster mode
Posted by Peter Schmitt <pe...@gmail.com>.
Hi Val,
thank you for checking it!
I've switched to JDK8 and the issue disappeared.
I've to test it with the JDK version we need to use in production.
It would be great to hear whether there is a different approach to reach
the same or if it could be a side-effect due to the used config:
//IgniteConfiguration + start as listed in the first mail
CacheConfiguration<String, CustomEntry> offheapCacheConfig = new
CacheConfiguration<String, CustomEntry>()
.setName(cacheName)
.setCacheMode(CacheMode.LOCAL)
.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED)
.setSwapEnabled(false)
.setBackups(0);
offheapCacheConfig.setOffHeapMaxMemory(...); //max. value loaded from a
config
LruEvictionPolicy evictionPolicy = new LruEvictionPolicy();
evictionPolicy.setMaxMemorySize(new
BigDecimal(offheapCacheConfig.getOffHeapMaxMemory() * 0.9).longValue());
IgniteCache<String, CustomEntry> cache =
ignite.createCache(offheapCacheConfig);
cache = cache.withExpiryPolicy(new AccessedExpiryPolicy(new
Duration(/*configured value*/)));
And to fill the cache the following code is used:
try (IgniteDataStreamer<String, CustomEntry> stream =
ignite.dataStreamer(cache.getName())) {
stream.allowOverwrite(true);
//use stream#addData
}
I hope that config makes sense and can't lead to the effect I described
initially.
Kind regards
Peter
2016-06-28 19:23 GMT+02:00 vkulichenko <va...@gmail.com>:
> Hi Peter,
>
> Your code works fine for me. Can you please attach the log file?
>
> -Val
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Non-cluster-mode-tp5959p5971.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Non-cluster mode
Posted by vkulichenko <va...@gmail.com>.
Hi Peter,
Your code works fine for me. Can you please attach the log file?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Non-cluster-mode-tp5959p5971.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.