You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Andrey Dolmatov <it...@gmail.com> on 2019/07/08 14:01:33 UTC
onheapCacheEnabled enormous heap consumption
We use simple replicated KV cache.
We try to upload 32 000 000 small records <Long, Long> to it (about 6Gb in
data region, persistance disabled). We load data using DataStreamer.
If we set onheapCacheEnabled=false, server node consumes heap about 500 Mb.
If we set onheapCacheEnabled=true, server node consumes heap about 6 Gb.
Why DataStreamer uses heap memory to load data? Why on-heap size is
unlimited (not just 100.000 records)? What default on-heap eviction policy?
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="cacheMode" value="REPLICATED"/>
<property name="onheapCacheEnabled" value="true"/>
<property name="copyOnRead" value="false"/>
Thanks!
Re: onheapCacheEnabled enormous heap consumption
Posted by Andrey Dolmatov <it...@gmail.com>.
Thanks! It was unclear for me.
On Sun, Jul 14, 2019, 10:59 AM Павлухин Иван <vo...@gmail.com> wrote:
> Andrey,
>
> Yes it is a little bit complicated for understanding.
>
> CacheConfiguration.evictionPolicy takes it roots back in history where
> there were neither offheap nor persisence, cache data was stored in
> java heap only. Also, as far as I know, today
> CacheConfiguration.evictionPolicy (as already mentioned) works only
> when onheap caching is enabled and an entry is still available in
> offheap after it was evicted from onheap. And it worth noting (correct
> me if I am wrong) that data is NOT buffered in onheap and then written
> to offheap on eviction but written to both places on each writing
> operation. So, I expect that onheap caching can speedup only read
> operations.
>
> DataRegionConfiguration.pageEvictionMode is a different thing (similar
> naming brings confusion) and it appeared with PageMemory. It cannot be
> configured on a cache level. It is applicable ONLY for data regions
> WITHOUT persistense and controls offheap pages eviction. If
> pageEvictionMode is not DataPageEvictionMode.DISABLED then in case of
> not enough memory (see also evictionThreshold) some page will be
> evicted (actually cleaned and reused).
>
> If persistence is enabled DataRegionConfiguration.pageEvictionMode has
> no effect (there should be a warning in Ignite node startup logs). For
> a persistent data regions there is an internal page replacement
> algorithm allowing to reuse offeap memory pages when some data need to
> be pulled from disk.
>
> пн, 8 июл. 2019 г. в 18:38, Ilya Kasnacheev <il...@gmail.com>:
> >
> > Hello!
> >
> > Data is always written to persistence immediately (via WAL). You can
> control eviction of offheap with evictionThreshold and pageEvictionMode
> settings of DataRegionConfiguration.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пн, 8 июл. 2019 г. в 17:50, Andrey Dolmatov <it...@gmail.com>:
> >>
> >> When data overfit dataRegion max size, so no more available offheap
> space, then data goes to persistence. So, what option controls how data
> pages should be evicted from offheap to persistence.
> >>
> >> On Mon, Jul 8, 2019, 5:33 PM Ilya Kasnacheev <il...@gmail.com>
> wrote:
> >>>
> >>> Hello!
> >>>
> >>> Data is always stored in offheap. Eviction strictly controls onheap
> cache. Once data is evicted from onheap it is available in offheap.
> >>>
> >>> Regards,
> >>> --
> >>> Ilya Kasnacheev
> >>>
> >>>
> >>> пн, 8 июл. 2019 г. в 17:31, Andrey Dolmatov <it...@gmail.com>:
> >>>>
> >>>> We plan to use persistence in production. I didn't understand,
> CacheConfiguration.EvictionPolicy specify heap->offheap eviction,
> offheap->persistence eviction or both. It's not clear for me.
> >>>>
> >>>> On Mon, Jul 8, 2019, 5:19 PM Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com> wrote:
> >>>>>
> >>>>> Hello!
> >>>>>
> >>>>> Oops, I was wrong. This is indeed the wrong setting.
> >>>>>
> >>>>> Have you tried specifying evictionPolicy? I think it is the one that
> controls eviction from onheap cache. You can put a LruEvictionPolicy of 100
> 000 here, for example.
> >>>>>
> >>>>> Regards,
> >>>>> --
> >>>>> Ilya Kasnacheev
> >>>>>
> >>>>>
> >>>>> пн, 8 июл. 2019 г. в 17:09, Andrey Dolmatov <it...@gmail.com>:
> >>>>>>
> >>>>>> No, because we didnt specify QueryEntity.
> >>>>>> Does onheapCacheEnabled uses for SQL only?
> >>>>>> What default value for sqlOnheapCacheMaxSize?
> >>>>>>
> >>>>>> пн, 8 июл. 2019 г. в 17:05, Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com>:
> >>>>>>>
> >>>>>>> Hello!
> >>>>>>>
> >>>>>>> Have you tried also specifying sqlOnheapCacheMaxSize? You can
> specify 100 000 if you like.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> --
> >>>>>>> Ilya Kasnacheev
> >>>>>>>
> >>>>>>>
> >>>>>>> пн, 8 июл. 2019 г. в 17:01, Andrey Dolmatov <itdolmatov@gmail.com
> >:
> >>>>>>>>
> >>>>>>>> We use simple replicated KV cache.
> >>>>>>>> We try to upload 32 000 000 small records <Long, Long> to it
> (about 6Gb in data region, persistance disabled). We load data using
> DataStreamer.
> >>>>>>>>
> >>>>>>>> If we set onheapCacheEnabled=false, server node consumes heap
> about 500 Mb.
> >>>>>>>> If we set onheapCacheEnabled=true, server node consumes heap
> about 6 Gb.
> >>>>>>>>
> >>>>>>>> Why DataStreamer uses heap memory to load data? Why on-heap size
> is unlimited (not just 100.000 records)? What default on-heap eviction
> policy?
> >>>>>>>>
> >>>>>>>> <bean class="org.apache.ignite.configuration.CacheConfiguration">
> >>>>>>>> <property name="cacheMode"
> value="REPLICATED"/>
> >>>>>>>> <property name="onheapCacheEnabled"
> value="true"/>
> >>>>>>>> <property name="copyOnRead" value="false"/>
> >>>>>>>>
> >>>>>>>> Thanks!
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>
Re: onheapCacheEnabled enormous heap consumption
Posted by Majid Salimi <ma...@gmail.com>.
Unsubscribe me
On Sun, Jul 14, 2019 at 12:30 PM Павлухин Иван <vo...@gmail.com> wrote:
> Andrey,
>
> Yes it is a little bit complicated for understanding.
>
> CacheConfiguration.evictionPolicy takes it roots back in history where
> there were neither offheap nor persisence, cache data was stored in
> java heap only. Also, as far as I know, today
> CacheConfiguration.evictionPolicy (as already mentioned) works only
> when onheap caching is enabled and an entry is still available in
> offheap after it was evicted from onheap. And it worth noting (correct
> me if I am wrong) that data is NOT buffered in onheap and then written
> to offheap on eviction but written to both places on each writing
> operation. So, I expect that onheap caching can speedup only read
> operations.
>
> DataRegionConfiguration.pageEvictionMode is a different thing (similar
> naming brings confusion) and it appeared with PageMemory. It cannot be
> configured on a cache level. It is applicable ONLY for data regions
> WITHOUT persistense and controls offheap pages eviction. If
> pageEvictionMode is not DataPageEvictionMode.DISABLED then in case of
> not enough memory (see also evictionThreshold) some page will be
> evicted (actually cleaned and reused).
>
> If persistence is enabled DataRegionConfiguration.pageEvictionMode has
> no effect (there should be a warning in Ignite node startup logs). For
> a persistent data regions there is an internal page replacement
> algorithm allowing to reuse offeap memory pages when some data need to
> be pulled from disk.
>
> пн, 8 июл. 2019 г. в 18:38, Ilya Kasnacheev <il...@gmail.com>:
> >
> > Hello!
> >
> > Data is always written to persistence immediately (via WAL). You can
> control eviction of offheap with evictionThreshold and pageEvictionMode
> settings of DataRegionConfiguration.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пн, 8 июл. 2019 г. в 17:50, Andrey Dolmatov <it...@gmail.com>:
> >>
> >> When data overfit dataRegion max size, so no more available offheap
> space, then data goes to persistence. So, what option controls how data
> pages should be evicted from offheap to persistence.
> >>
> >> On Mon, Jul 8, 2019, 5:33 PM Ilya Kasnacheev <il...@gmail.com>
> wrote:
> >>>
> >>> Hello!
> >>>
> >>> Data is always stored in offheap. Eviction strictly controls onheap
> cache. Once data is evicted from onheap it is available in offheap.
> >>>
> >>> Regards,
> >>> --
> >>> Ilya Kasnacheev
> >>>
> >>>
> >>> пн, 8 июл. 2019 г. в 17:31, Andrey Dolmatov <it...@gmail.com>:
> >>>>
> >>>> We plan to use persistence in production. I didn't understand,
> CacheConfiguration.EvictionPolicy specify heap->offheap eviction,
> offheap->persistence eviction or both. It's not clear for me.
> >>>>
> >>>> On Mon, Jul 8, 2019, 5:19 PM Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com> wrote:
> >>>>>
> >>>>> Hello!
> >>>>>
> >>>>> Oops, I was wrong. This is indeed the wrong setting.
> >>>>>
> >>>>> Have you tried specifying evictionPolicy? I think it is the one that
> controls eviction from onheap cache. You can put a LruEvictionPolicy of 100
> 000 here, for example.
> >>>>>
> >>>>> Regards,
> >>>>> --
> >>>>> Ilya Kasnacheev
> >>>>>
> >>>>>
> >>>>> пн, 8 июл. 2019 г. в 17:09, Andrey Dolmatov <it...@gmail.com>:
> >>>>>>
> >>>>>> No, because we didnt specify QueryEntity.
> >>>>>> Does onheapCacheEnabled uses for SQL only?
> >>>>>> What default value for sqlOnheapCacheMaxSize?
> >>>>>>
> >>>>>> пн, 8 июл. 2019 г. в 17:05, Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com>:
> >>>>>>>
> >>>>>>> Hello!
> >>>>>>>
> >>>>>>> Have you tried also specifying sqlOnheapCacheMaxSize? You can
> specify 100 000 if you like.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> --
> >>>>>>> Ilya Kasnacheev
> >>>>>>>
> >>>>>>>
> >>>>>>> пн, 8 июл. 2019 г. в 17:01, Andrey Dolmatov <itdolmatov@gmail.com
> >:
> >>>>>>>>
> >>>>>>>> We use simple replicated KV cache.
> >>>>>>>> We try to upload 32 000 000 small records <Long, Long> to it
> (about 6Gb in data region, persistance disabled). We load data using
> DataStreamer.
> >>>>>>>>
> >>>>>>>> If we set onheapCacheEnabled=false, server node consumes heap
> about 500 Mb.
> >>>>>>>> If we set onheapCacheEnabled=true, server node consumes heap
> about 6 Gb.
> >>>>>>>>
> >>>>>>>> Why DataStreamer uses heap memory to load data? Why on-heap size
> is unlimited (not just 100.000 records)? What default on-heap eviction
> policy?
> >>>>>>>>
> >>>>>>>> <bean class="org.apache.ignite.configuration.CacheConfiguration">
> >>>>>>>> <property name="cacheMode"
> value="REPLICATED"/>
> >>>>>>>> <property name="onheapCacheEnabled"
> value="true"/>
> >>>>>>>> <property name="copyOnRead" value="false"/>
> >>>>>>>>
> >>>>>>>> Thanks!
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>
--
*Regards,Majid Salimi BeniM.Sc. Student of Computer Engineering,Department
of Computer Science and Engineering & IT*
*Shiraz University*
Attachments area
Re: onheapCacheEnabled enormous heap consumption
Posted by Павлухин Иван <vo...@gmail.com>.
Andrey,
Yes it is a little bit complicated for understanding.
CacheConfiguration.evictionPolicy takes it roots back in history where
there were neither offheap nor persisence, cache data was stored in
java heap only. Also, as far as I know, today
CacheConfiguration.evictionPolicy (as already mentioned) works only
when onheap caching is enabled and an entry is still available in
offheap after it was evicted from onheap. And it worth noting (correct
me if I am wrong) that data is NOT buffered in onheap and then written
to offheap on eviction but written to both places on each writing
operation. So, I expect that onheap caching can speedup only read
operations.
DataRegionConfiguration.pageEvictionMode is a different thing (similar
naming brings confusion) and it appeared with PageMemory. It cannot be
configured on a cache level. It is applicable ONLY for data regions
WITHOUT persistense and controls offheap pages eviction. If
pageEvictionMode is not DataPageEvictionMode.DISABLED then in case of
not enough memory (see also evictionThreshold) some page will be
evicted (actually cleaned and reused).
If persistence is enabled DataRegionConfiguration.pageEvictionMode has
no effect (there should be a warning in Ignite node startup logs). For
a persistent data regions there is an internal page replacement
algorithm allowing to reuse offeap memory pages when some data need to
be pulled from disk.
пн, 8 июл. 2019 г. в 18:38, Ilya Kasnacheev <il...@gmail.com>:
>
> Hello!
>
> Data is always written to persistence immediately (via WAL). You can control eviction of offheap with evictionThreshold and pageEvictionMode settings of DataRegionConfiguration.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 8 июл. 2019 г. в 17:50, Andrey Dolmatov <it...@gmail.com>:
>>
>> When data overfit dataRegion max size, so no more available offheap space, then data goes to persistence. So, what option controls how data pages should be evicted from offheap to persistence.
>>
>> On Mon, Jul 8, 2019, 5:33 PM Ilya Kasnacheev <il...@gmail.com> wrote:
>>>
>>> Hello!
>>>
>>> Data is always stored in offheap. Eviction strictly controls onheap cache. Once data is evicted from onheap it is available in offheap.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пн, 8 июл. 2019 г. в 17:31, Andrey Dolmatov <it...@gmail.com>:
>>>>
>>>> We plan to use persistence in production. I didn't understand, CacheConfiguration.EvictionPolicy specify heap->offheap eviction, offheap->persistence eviction or both. It's not clear for me.
>>>>
>>>> On Mon, Jul 8, 2019, 5:19 PM Ilya Kasnacheev <il...@gmail.com> wrote:
>>>>>
>>>>> Hello!
>>>>>
>>>>> Oops, I was wrong. This is indeed the wrong setting.
>>>>>
>>>>> Have you tried specifying evictionPolicy? I think it is the one that controls eviction from onheap cache. You can put a LruEvictionPolicy of 100 000 here, for example.
>>>>>
>>>>> Regards,
>>>>> --
>>>>> Ilya Kasnacheev
>>>>>
>>>>>
>>>>> пн, 8 июл. 2019 г. в 17:09, Andrey Dolmatov <it...@gmail.com>:
>>>>>>
>>>>>> No, because we didnt specify QueryEntity.
>>>>>> Does onheapCacheEnabled uses for SQL only?
>>>>>> What default value for sqlOnheapCacheMaxSize?
>>>>>>
>>>>>> пн, 8 июл. 2019 г. в 17:05, Ilya Kasnacheev <il...@gmail.com>:
>>>>>>>
>>>>>>> Hello!
>>>>>>>
>>>>>>> Have you tried also specifying sqlOnheapCacheMaxSize? You can specify 100 000 if you like.
>>>>>>>
>>>>>>> Regards,
>>>>>>> --
>>>>>>> Ilya Kasnacheev
>>>>>>>
>>>>>>>
>>>>>>> пн, 8 июл. 2019 г. в 17:01, Andrey Dolmatov <it...@gmail.com>:
>>>>>>>>
>>>>>>>> We use simple replicated KV cache.
>>>>>>>> We try to upload 32 000 000 small records <Long, Long> to it (about 6Gb in data region, persistance disabled). We load data using DataStreamer.
>>>>>>>>
>>>>>>>> If we set onheapCacheEnabled=false, server node consumes heap about 500 Mb.
>>>>>>>> If we set onheapCacheEnabled=true, server node consumes heap about 6 Gb.
>>>>>>>>
>>>>>>>> Why DataStreamer uses heap memory to load data? Why on-heap size is unlimited (not just 100.000 records)? What default on-heap eviction policy?
>>>>>>>>
>>>>>>>> <bean class="org.apache.ignite.configuration.CacheConfiguration">
>>>>>>>> <property name="cacheMode" value="REPLICATED"/>
>>>>>>>> <property name="onheapCacheEnabled" value="true"/>
>>>>>>>> <property name="copyOnRead" value="false"/>
>>>>>>>>
>>>>>>>> Thanks!
--
Best regards,
Ivan Pavlukhin
Re: onheapCacheEnabled enormous heap consumption
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Data is always written to persistence immediately (via WAL). You can
control eviction of offheap with evictionThreshold and pageEvictionMode
settings of DataRegionConfiguration.
Regards,
--
Ilya Kasnacheev
пн, 8 июл. 2019 г. в 17:50, Andrey Dolmatov <it...@gmail.com>:
> When data overfit dataRegion max size, so no more available offheap space,
> then data goes to persistence. So, what option controls how data pages
> should be evicted from offheap to persistence.
>
> On Mon, Jul 8, 2019, 5:33 PM Ilya Kasnacheev <il...@gmail.com>
> wrote:
>
>> Hello!
>>
>> Data is always stored in offheap. Eviction strictly controls onheap
>> cache. Once data is evicted from onheap it is available in offheap.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 8 июл. 2019 г. в 17:31, Andrey Dolmatov <it...@gmail.com>:
>>
>>> We plan to use persistence in production. I didn't understand,
>>> CacheConfiguration.EvictionPolicy specify heap->offheap eviction,
>>> offheap->persistence eviction or both. It's not clear for me.
>>>
>>> On Mon, Jul 8, 2019, 5:19 PM Ilya Kasnacheev <il...@gmail.com>
>>> wrote:
>>>
>>>> Hello!
>>>>
>>>> Oops, I was wrong. This is indeed the wrong setting.
>>>>
>>>> Have you tried specifying evictionPolicy? I think it is the one that
>>>> controls eviction from onheap cache. You can put a LruEvictionPolicy of 100
>>>> 000 here, for example.
>>>>
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>>
>>>>
>>>> пн, 8 июл. 2019 г. в 17:09, Andrey Dolmatov <it...@gmail.com>:
>>>>
>>>>> No, because we didnt specify QueryEntity.
>>>>> Does onheapCacheEnabled uses for SQL only?
>>>>> What default value for sqlOnheapCacheMaxSize?
>>>>>
>>>>> пн, 8 июл. 2019 г. в 17:05, Ilya Kasnacheev <ilya.kasnacheev@gmail.com
>>>>> >:
>>>>>
>>>>>> Hello!
>>>>>>
>>>>>> Have you tried also specifying sqlOnheapCacheMaxSize? You can specify
>>>>>> 100 000 if you like.
>>>>>>
>>>>>> Regards,
>>>>>> --
>>>>>> Ilya Kasnacheev
>>>>>>
>>>>>>
>>>>>> пн, 8 июл. 2019 г. в 17:01, Andrey Dolmatov <it...@gmail.com>:
>>>>>>
>>>>>>> We use simple replicated KV cache.
>>>>>>> We try to upload 32 000 000 small records <Long, Long> to it (about
>>>>>>> 6Gb in data region, persistance disabled). We load data using DataStreamer.
>>>>>>>
>>>>>>> If we set onheapCacheEnabled=false, server node consumes heap about
>>>>>>> 500 Mb.
>>>>>>> If we set onheapCacheEnabled=true, server node consumes heap about 6
>>>>>>> Gb.
>>>>>>>
>>>>>>> Why DataStreamer uses heap memory to load data? Why on-heap size is
>>>>>>> unlimited (not just 100.000 records)? What default on-heap eviction policy?
>>>>>>>
>>>>>>> <bean class="org.apache.ignite.configuration.CacheConfiguration">
>>>>>>>
>>>>>>> <property name="cacheMode" value="REPLICATED"/>
>>>>>>> <property name="onheapCacheEnabled"
>>>>>>> value="true"/>
>>>>>>> <property name="copyOnRead" value="false"/>
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>
Re: onheapCacheEnabled enormous heap consumption
Posted by Andrey Dolmatov <it...@gmail.com>.
When data overfit dataRegion max size, so no more available offheap space,
then data goes to persistence. So, what option controls how data pages
should be evicted from offheap to persistence.
On Mon, Jul 8, 2019, 5:33 PM Ilya Kasnacheev <il...@gmail.com>
wrote:
> Hello!
>
> Data is always stored in offheap. Eviction strictly controls onheap cache.
> Once data is evicted from onheap it is available in offheap.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 8 июл. 2019 г. в 17:31, Andrey Dolmatov <it...@gmail.com>:
>
>> We plan to use persistence in production. I didn't understand,
>> CacheConfiguration.EvictionPolicy specify heap->offheap eviction,
>> offheap->persistence eviction or both. It's not clear for me.
>>
>> On Mon, Jul 8, 2019, 5:19 PM Ilya Kasnacheev <il...@gmail.com>
>> wrote:
>>
>>> Hello!
>>>
>>> Oops, I was wrong. This is indeed the wrong setting.
>>>
>>> Have you tried specifying evictionPolicy? I think it is the one that
>>> controls eviction from onheap cache. You can put a LruEvictionPolicy of 100
>>> 000 here, for example.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пн, 8 июл. 2019 г. в 17:09, Andrey Dolmatov <it...@gmail.com>:
>>>
>>>> No, because we didnt specify QueryEntity.
>>>> Does onheapCacheEnabled uses for SQL only?
>>>> What default value for sqlOnheapCacheMaxSize?
>>>>
>>>> пн, 8 июл. 2019 г. в 17:05, Ilya Kasnacheev <ilya.kasnacheev@gmail.com
>>>> >:
>>>>
>>>>> Hello!
>>>>>
>>>>> Have you tried also specifying sqlOnheapCacheMaxSize? You can specify
>>>>> 100 000 if you like.
>>>>>
>>>>> Regards,
>>>>> --
>>>>> Ilya Kasnacheev
>>>>>
>>>>>
>>>>> пн, 8 июл. 2019 г. в 17:01, Andrey Dolmatov <it...@gmail.com>:
>>>>>
>>>>>> We use simple replicated KV cache.
>>>>>> We try to upload 32 000 000 small records <Long, Long> to it (about
>>>>>> 6Gb in data region, persistance disabled). We load data using DataStreamer.
>>>>>>
>>>>>> If we set onheapCacheEnabled=false, server node consumes heap about
>>>>>> 500 Mb.
>>>>>> If we set onheapCacheEnabled=true, server node consumes heap about 6
>>>>>> Gb.
>>>>>>
>>>>>> Why DataStreamer uses heap memory to load data? Why on-heap size is
>>>>>> unlimited (not just 100.000 records)? What default on-heap eviction policy?
>>>>>>
>>>>>> <bean class="org.apache.ignite.configuration.CacheConfiguration">
>>>>>>
>>>>>> <property name="cacheMode" value="REPLICATED"/>
>>>>>> <property name="onheapCacheEnabled" value="true"/>
>>>>>> <property name="copyOnRead" value="false"/>
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>
Re: onheapCacheEnabled enormous heap consumption
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Data is always stored in offheap. Eviction strictly controls onheap cache.
Once data is evicted from onheap it is available in offheap.
Regards,
--
Ilya Kasnacheev
пн, 8 июл. 2019 г. в 17:31, Andrey Dolmatov <it...@gmail.com>:
> We plan to use persistence in production. I didn't understand,
> CacheConfiguration.EvictionPolicy specify heap->offheap eviction,
> offheap->persistence eviction or both. It's not clear for me.
>
> On Mon, Jul 8, 2019, 5:19 PM Ilya Kasnacheev <il...@gmail.com>
> wrote:
>
>> Hello!
>>
>> Oops, I was wrong. This is indeed the wrong setting.
>>
>> Have you tried specifying evictionPolicy? I think it is the one that
>> controls eviction from onheap cache. You can put a LruEvictionPolicy of 100
>> 000 here, for example.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 8 июл. 2019 г. в 17:09, Andrey Dolmatov <it...@gmail.com>:
>>
>>> No, because we didnt specify QueryEntity.
>>> Does onheapCacheEnabled uses for SQL only?
>>> What default value for sqlOnheapCacheMaxSize?
>>>
>>> пн, 8 июл. 2019 г. в 17:05, Ilya Kasnacheev <il...@gmail.com>:
>>>
>>>> Hello!
>>>>
>>>> Have you tried also specifying sqlOnheapCacheMaxSize? You can specify
>>>> 100 000 if you like.
>>>>
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>>
>>>>
>>>> пн, 8 июл. 2019 г. в 17:01, Andrey Dolmatov <it...@gmail.com>:
>>>>
>>>>> We use simple replicated KV cache.
>>>>> We try to upload 32 000 000 small records <Long, Long> to it (about
>>>>> 6Gb in data region, persistance disabled). We load data using DataStreamer.
>>>>>
>>>>> If we set onheapCacheEnabled=false, server node consumes heap about
>>>>> 500 Mb.
>>>>> If we set onheapCacheEnabled=true, server node consumes heap about 6
>>>>> Gb.
>>>>>
>>>>> Why DataStreamer uses heap memory to load data? Why on-heap size is
>>>>> unlimited (not just 100.000 records)? What default on-heap eviction policy?
>>>>>
>>>>> <bean class="org.apache.ignite.configuration.CacheConfiguration">
>>>>>
>>>>> <property name="cacheMode" value="REPLICATED"/>
>>>>> <property name="onheapCacheEnabled" value="true"/>
>>>>> <property name="copyOnRead" value="false"/>
>>>>>
>>>>> Thanks!
>>>>>
>>>>
Re: onheapCacheEnabled enormous heap consumption
Posted by Andrey Dolmatov <it...@gmail.com>.
We plan to use persistence in production. I didn't understand,
CacheConfiguration.EvictionPolicy specify heap->offheap eviction,
offheap->persistence eviction or both. It's not clear for me.
On Mon, Jul 8, 2019, 5:19 PM Ilya Kasnacheev <il...@gmail.com>
wrote:
> Hello!
>
> Oops, I was wrong. This is indeed the wrong setting.
>
> Have you tried specifying evictionPolicy? I think it is the one that
> controls eviction from onheap cache. You can put a LruEvictionPolicy of 100
> 000 here, for example.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 8 июл. 2019 г. в 17:09, Andrey Dolmatov <it...@gmail.com>:
>
>> No, because we didnt specify QueryEntity.
>> Does onheapCacheEnabled uses for SQL only?
>> What default value for sqlOnheapCacheMaxSize?
>>
>> пн, 8 июл. 2019 г. в 17:05, Ilya Kasnacheev <il...@gmail.com>:
>>
>>> Hello!
>>>
>>> Have you tried also specifying sqlOnheapCacheMaxSize? You can specify
>>> 100 000 if you like.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пн, 8 июл. 2019 г. в 17:01, Andrey Dolmatov <it...@gmail.com>:
>>>
>>>> We use simple replicated KV cache.
>>>> We try to upload 32 000 000 small records <Long, Long> to it (about 6Gb
>>>> in data region, persistance disabled). We load data using DataStreamer.
>>>>
>>>> If we set onheapCacheEnabled=false, server node consumes heap about 500
>>>> Mb.
>>>> If we set onheapCacheEnabled=true, server node consumes heap about 6 Gb.
>>>>
>>>> Why DataStreamer uses heap memory to load data? Why on-heap size is
>>>> unlimited (not just 100.000 records)? What default on-heap eviction policy?
>>>>
>>>> <bean class="org.apache.ignite.configuration.CacheConfiguration">
>>>>
>>>> <property name="cacheMode" value="REPLICATED"/>
>>>> <property name="onheapCacheEnabled" value="true"/>
>>>> <property name="copyOnRead" value="false"/>
>>>>
>>>> Thanks!
>>>>
>>>
Re: onheapCacheEnabled enormous heap consumption
Posted by Andrey Dolmatov <it...@gmail.com>.
No, we didn't specify EvictionPolicy. I didn't find does CacheConfiguration
have default EvictionPolicy?
On Mon, Jul 8, 2019, 5:19 PM Ilya Kasnacheev <il...@gmail.com>
wrote:
> Hello!
>
> Oops, I was wrong. This is indeed the wrong setting.
>
> Have you tried specifying evictionPolicy? I think it is the one that
> controls eviction from onheap cache. You can put a LruEvictionPolicy of 100
> 000 here, for example.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 8 июл. 2019 г. в 17:09, Andrey Dolmatov <it...@gmail.com>:
>
>> No, because we didnt specify QueryEntity.
>> Does onheapCacheEnabled uses for SQL only?
>> What default value for sqlOnheapCacheMaxSize?
>>
>> пн, 8 июл. 2019 г. в 17:05, Ilya Kasnacheev <il...@gmail.com>:
>>
>>> Hello!
>>>
>>> Have you tried also specifying sqlOnheapCacheMaxSize? You can specify
>>> 100 000 if you like.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пн, 8 июл. 2019 г. в 17:01, Andrey Dolmatov <it...@gmail.com>:
>>>
>>>> We use simple replicated KV cache.
>>>> We try to upload 32 000 000 small records <Long, Long> to it (about 6Gb
>>>> in data region, persistance disabled). We load data using DataStreamer.
>>>>
>>>> If we set onheapCacheEnabled=false, server node consumes heap about 500
>>>> Mb.
>>>> If we set onheapCacheEnabled=true, server node consumes heap about 6 Gb.
>>>>
>>>> Why DataStreamer uses heap memory to load data? Why on-heap size is
>>>> unlimited (not just 100.000 records)? What default on-heap eviction policy?
>>>>
>>>> <bean class="org.apache.ignite.configuration.CacheConfiguration">
>>>>
>>>> <property name="cacheMode" value="REPLICATED"/>
>>>> <property name="onheapCacheEnabled" value="true"/>
>>>> <property name="copyOnRead" value="false"/>
>>>>
>>>> Thanks!
>>>>
>>>
Re: onheapCacheEnabled enormous heap consumption
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Oops, I was wrong. This is indeed the wrong setting.
Have you tried specifying evictionPolicy? I think it is the one that
controls eviction from onheap cache. You can put a LruEvictionPolicy of 100
000 here, for example.
Regards,
--
Ilya Kasnacheev
пн, 8 июл. 2019 г. в 17:09, Andrey Dolmatov <it...@gmail.com>:
> No, because we didnt specify QueryEntity.
> Does onheapCacheEnabled uses for SQL only?
> What default value for sqlOnheapCacheMaxSize?
>
> пн, 8 июл. 2019 г. в 17:05, Ilya Kasnacheev <il...@gmail.com>:
>
>> Hello!
>>
>> Have you tried also specifying sqlOnheapCacheMaxSize? You can specify 100
>> 000 if you like.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 8 июл. 2019 г. в 17:01, Andrey Dolmatov <it...@gmail.com>:
>>
>>> We use simple replicated KV cache.
>>> We try to upload 32 000 000 small records <Long, Long> to it (about 6Gb
>>> in data region, persistance disabled). We load data using DataStreamer.
>>>
>>> If we set onheapCacheEnabled=false, server node consumes heap about 500
>>> Mb.
>>> If we set onheapCacheEnabled=true, server node consumes heap about 6 Gb.
>>>
>>> Why DataStreamer uses heap memory to load data? Why on-heap size is
>>> unlimited (not just 100.000 records)? What default on-heap eviction policy?
>>>
>>> <bean class="org.apache.ignite.configuration.CacheConfiguration">
>>>
>>> <property name="cacheMode" value="REPLICATED"/>
>>> <property name="onheapCacheEnabled" value="true"/>
>>> <property name="copyOnRead" value="false"/>
>>>
>>> Thanks!
>>>
>>
Re: onheapCacheEnabled enormous heap consumption
Posted by Andrey Dolmatov <it...@gmail.com>.
No, because we didnt specify QueryEntity.
Does onheapCacheEnabled uses for SQL only?
What default value for sqlOnheapCacheMaxSize?
пн, 8 июл. 2019 г. в 17:05, Ilya Kasnacheev <il...@gmail.com>:
> Hello!
>
> Have you tried also specifying sqlOnheapCacheMaxSize? You can specify 100
> 000 if you like.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 8 июл. 2019 г. в 17:01, Andrey Dolmatov <it...@gmail.com>:
>
>> We use simple replicated KV cache.
>> We try to upload 32 000 000 small records <Long, Long> to it (about 6Gb
>> in data region, persistance disabled). We load data using DataStreamer.
>>
>> If we set onheapCacheEnabled=false, server node consumes heap about 500
>> Mb.
>> If we set onheapCacheEnabled=true, server node consumes heap about 6 Gb.
>>
>> Why DataStreamer uses heap memory to load data? Why on-heap size is
>> unlimited (not just 100.000 records)? What default on-heap eviction policy?
>>
>> <bean class="org.apache.ignite.configuration.CacheConfiguration">
>>
>> <property name="cacheMode" value="REPLICATED"/>
>> <property name="onheapCacheEnabled" value="true"/>
>> <property name="copyOnRead" value="false"/>
>>
>> Thanks!
>>
>
Re: onheapCacheEnabled enormous heap consumption
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Have you tried also specifying sqlOnheapCacheMaxSize? You can specify 100
000 if you like.
Regards,
--
Ilya Kasnacheev
пн, 8 июл. 2019 г. в 17:01, Andrey Dolmatov <it...@gmail.com>:
> We use simple replicated KV cache.
> We try to upload 32 000 000 small records <Long, Long> to it (about 6Gb in
> data region, persistance disabled). We load data using DataStreamer.
>
> If we set onheapCacheEnabled=false, server node consumes heap about 500 Mb.
> If we set onheapCacheEnabled=true, server node consumes heap about 6 Gb.
>
> Why DataStreamer uses heap memory to load data? Why on-heap size is
> unlimited (not just 100.000 records)? What default on-heap eviction policy?
>
> <bean class="org.apache.ignite.configuration.CacheConfiguration">
>
> <property name="cacheMode" value="REPLICATED"/>
> <property name="onheapCacheEnabled" value="true"/>
> <property name="copyOnRead" value="false"/>
>
> Thanks!
>