You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@ignite.apache.org by charly <cd...@appnexus.com> on 2017/06/21 23:11:10 UTC

Small value for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE

Hey everyone,

We use Ignite 1.8 and see a difference in heap used when lowering the value
for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE. I'm not sure to understand what
it's going to happen if we decrease that value on production. Could someone
clear that up for me please? From what I understand, deleted entries go in
that queue for cleaning, where one thread per cache is in charge of cleaning
the queue. But what if the queue is full? I might have missed that
information in the documentation, my bad if I did.

Maybe on the same subject, is it going to slow down the free up of the heap
after cache.destroy()? We have a use-case where we delete 4 caches at the
same time containing ~500k entries heavily indexed. When doing so, it takes
few minutes for the heap to decrease as you can see here
http://imgur.com/a/kPsSc . We would not want to extend the time to free up
memory even more.

More information about our setup:
- on heap (we'll eventually move to off heap but back then we could not
make Ignite 1.8 to free up memory at all with our version of glibc. It would
work with jmalloc but that was not permitted on production at that time).
- no backup
- atomicity mode: atomic
- cache mode: partitioned
- cluster of 4 nodes
- jvm options: -J-DIGNITE_LONG_OPERATIONS_DUMP_TIMEOUT=300000 -J-Xms12g
-J-Xmx12g -J-server -J-XX:+UseParNewGC -J-XX:+UseConcMarkSweepGC
-J-XX:+UseTLAB -J-XX:NewSize=128m -J-XX:MaxNewSize=128m
-J-XX:MaxTenuringThreshold=0 -J-XX:SurvivorRatio=1024
-J-XX:+UseCMSInitiatingOccupancyOnly -J-XX:CMSInitiatingOccupancyFraction=10
-J-XX:MaxGCPauseMillis=1000 -J-XX:InitiatingHeapOccupancyPercent=10
-J-XX:+UseCompressedOops -J-XX:ParallelGCThreads=8 -J-XX:ConcGCThreads=8
-J-XX:+PrintGCDetails -J-XX:+PrintGCTimeStamps -J-XX:+PrintGCDateStamps
-J-XX:+UseGCLogFileRotation -J-XX:NumberOfGCLogFiles=10
-J-XX:GCLogFileSize=100M -J-Xloggc:/tmp/ignite-gc.log
-J-Dcom.sun.management.jmxremote.port=49130

Thanks for your help,
Charly

--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Small-value-for-IGNITE-ATOMIC-CACHE-DELETE-HISTORY-SIZE-tp14037.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

RE: Small value for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE

Posted by charly <cd...@appnexus.com>.

Thanks for all your answers Ilya!
________________________________________
From: Ilya Lantukh [via Apache Ignite Users] [ml+s70518n14271h42@n6.nabble.com]
Sent: Monday, July 03, 2017 9:20 AM
To: Charly Dessart
Subject: Re: Small value for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE

Yes, such races during pre-loading are possible even if you configured 0 backups.

On Fri, Jun 30, 2017 at 7:31 PM, charly <[hidden email]</user/SendEmail.jtp?type=node&node=14271&i=0>> wrote:
It helps a lot. Thanks for the detailed example. I have a last question. Would it still be a potential issue to lower the queue size if we don't have any backup? Like if a new node pre-loads from the primary, any new changes will also be populated to the soon-to-be new primary?

Sent from my iPhone

On Jun 30, 2017, at 11:27 AM, Ilya Lantukh [via Apache Ignite Users] <[hidden email]<http:///user/SendEmail.jtp?type=node&node=14183&i=0>> wrote:

Hi Charly,

There is no special thread in charge of clearing the queue. Entry is cleared when you try to add new entry to the queue, but it is already full. So, once queue becomes full, it will always stay full, and lowering queue size obviously reduces heap consumption. If cache is destroyed, the queue should be cleared immediately.

However, if queue size is too low, it may lead to data inconsistency on unstable topology.

Here is an example:
Node A just joined the cluster, became primary node for some partitions and started preloading from node B, which is backup. Node B sends value V for key K, but before it was received and processed on node A user executed cache.remove(K).
- If queue size is 0, node A now doesn't have any information that entry was just removed and has to store (K, V) pair from node B. On the other hand, node B will remove K and now primary and backup nodes have different data.
- If queue size is N > 0, node A will save a tombstone entry for K. It will have all necessary information to understand that (K, V) pair from node B is older than current value and don't store it. Tombstone will be cleared after N other keys are removed.

Hope this helps.

On Thu, Jun 22, 2017 at 2:11 AM, charly <[hidden email]<http:///user/SendEmail.jtp?type=node&node=14182&i=0>> wrote:
Hey everyone,

We use Ignite 1.8 and see a difference in heap used when lowering the value
for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE. I'm not sure to understand what
it's going to happen if we decrease that value on production. Could someone
clear that up for me please? From what I understand, deleted entries go in
that queue for cleaning, where one thread per cache is in charge of cleaning
the queue. But what if the queue is full? I might have missed that
information in the documentation, my bad if I did.

Maybe on the same subject, is it going to slow down the free up of the heap
after cache.destroy()? We have a use-case where we delete 4 caches at the
same time containing ~500k entries heavily indexed. When doing so, it takes
few minutes for the heap to decrease as you can see here
http://imgur.com/a/kPsSc . We would not want to extend the time to free up
memory even more.

More information about our setup:
 - on heap (we'll eventually move to off heap but back then we could not
make Ignite 1.8 to free up memory at all with our version of glibc. It would
work with jmalloc but that was not permitted on production at that time).
 - no backup
 - atomicity mode: atomic
 - cache mode: partitioned
 - cluster of 4 nodes
 - jvm options: -J-DIGNITE_LONG_OPERATIONS_DUMP_TIMEOUT=300000 -J-Xms12g
-J-Xmx12g -J-server -J-XX:+UseParNewGC -J-XX:+UseConcMarkSweepGC
-J-XX:+UseTLAB -J-XX:NewSize=128m -J-XX:MaxNewSize=128m
-J-XX:MaxTenuringThreshold=0 -J-XX:SurvivorRatio=1024
-J-XX:+UseCMSInitiatingOccupancyOnly -J-XX:CMSInitiatingOccupancyFraction=10
-J-XX:MaxGCPauseMillis=1000 -J-XX:InitiatingHeapOccupancyPercent=10
-J-XX:+UseCompressedOops -J-XX:ParallelGCThreads=8 -J-XX:ConcGCThreads=8
-J-XX:+PrintGCDetails -J-XX:+PrintGCTimeStamps -J-XX:+PrintGCDateStamps
-J-XX:+UseGCLogFileRotation -J-XX:NumberOfGCLogFiles=10
-J-XX:GCLogFileSize=100M -J-Xloggc:/tmp/ignite-gc.log
-J-Dcom.sun.management.jmxremote.port=49130

Thanks for your help,
Charly

--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Small-value-for-IGNITE-ATOMIC-CACHE-DELETE-HISTORY-SIZE-tp14037.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com<http://Nabble.com>.

--
Best regards,
Ilya

________________________________
If you reply to this email, your message will be added to the discussion below:
http://apache-ignite-users.70518.x6.nabble.com/Small-value-for-IGNITE-ATOMIC-CACHE-DELETE-HISTORY-SIZE-tp14037p14182.html
To unsubscribe from Small value for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE, click here.
NAML<http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

________________________________
View this message in context: Re: Small value for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE<http://apache-ignite-users.70518.x6.nabble.com/Small-value-for-IGNITE-ATOMIC-CACHE-DELETE-HISTORY-SIZE-tp14037p14183.html>

Sent from the Apache Ignite Users mailing list archive<http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.

--
Best regards,
Ilya

________________________________
If you reply to this email, your message will be added to the discussion below:
http://apache-ignite-users.70518.x6.nabble.com/Small-value-for-IGNITE-ATOMIC-CACHE-DELETE-HISTORY-SIZE-tp14037p14271.html
To unsubscribe from Small value for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE, click here<http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=14037&code=Y2Rlc3NhcnRAYXBwbmV4dXMuY29tfDE0MDM3fDE3Mzc3NzQ4OTA=>.
NAML<http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Small-value-for-IGNITE-ATOMIC-CACHE-DELETE-HISTORY-SIZE-tp14037p14399.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Small value for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE

Posted by Ilya Lantukh <il...@gridgain.com>.

Yes, such races during pre-loading are possible even if you configured 0
backups.

On Fri, Jun 30, 2017 at 7:31 PM, charly <cd...@appnexus.com> wrote:

> It helps a lot. Thanks for the detailed example. I have a last question.
> Would it still be a potential issue to lower the queue size if we don't
> have any backup? Like if a new node pre-loads from the primary, any new
> changes will also be populated to the soon-to-be new primary?
>
> Sent from my iPhone
>
> On Jun 30, 2017, at 11:27 AM, Ilya Lantukh [via Apache Ignite Users] <[hidden
> email] <http:///user/SendEmail.jtp?type=node&node=14183&i=0>> wrote:
>
> Hi Charly,
>
> There is no special thread in charge of clearing the queue. Entry is
> cleared when you try to add new entry to the queue, but it is already full.
> So, once queue becomes full, it will always stay full, and lowering queue
> size obviously reduces heap consumption. If cache is destroyed, the queue
> should be cleared immediately.
>
> However, if queue size is too low, it may lead to data inconsistency on
> unstable topology.
>
> Here is an example:
> Node A just joined the cluster, became primary node for some partitions
> and started preloading from node B, which is backup. Node B sends value V
> for key K, but before it was received and processed on node A user executed
> cache.remove(K).
> - If queue size is 0, node A now doesn't have any information that entry
> was just removed and has to store (K, V) pair from node B. On the other
> hand, node B will remove K and now primary and backup nodes have different
> data.
> - If queue size is N > 0, node A will save a tombstone entry for K. It
> will have all necessary information to understand that (K, V) pair from
> node B is older than current value and don't store it. Tombstone will be
> cleared after N other keys are removed.
>
> Hope this helps.
>
>
> On Thu, Jun 22, 2017 at 2:11 AM, charly <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=14182&i=0>> wrote:
>
>> Hey everyone,
>>
>> We use Ignite 1.8 and see a difference in heap used when lowering the
>> value
>> for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE. I'm not sure to understand
>> what
>> it's going to happen if we decrease that value on production. Could
>> someone
>> clear that up for me please? From what I understand, deleted entries go in
>> that queue for cleaning, where one thread per cache is in charge of
>> cleaning
>> the queue. But what if the queue is full? I might have missed that
>> information in the documentation, my bad if I did.
>>
>> Maybe on the same subject, is it going to slow down the free up of the
>> heap
>> after cache.destroy()? We have a use-case where we delete 4 caches at the
>> same time containing ~500k entries heavily indexed. When doing so, it
>> takes
>> few minutes for the heap to decrease as you can see here
>> http://imgur.com/a/kPsSc . We would not want to extend the time to free
>> up
>> memory even more.
>>
>> More information about our setup:
>>  - on heap (we'll eventually move to off heap but back then we could not
>> make Ignite 1.8 to free up memory at all with our version of glibc. It
>> would
>> work with jmalloc but that was not permitted on production at that time).
>>  - no backup
>>  - atomicity mode: atomic
>>  - cache mode: partitioned
>>  - cluster of 4 nodes
>>  - jvm options: -J-DIGNITE_LONG_OPERATIONS_DUMP_TIMEOUT=300000 -J-Xms12g
>> -J-Xmx12g -J-server -J-XX:+UseParNewGC -J-XX:+UseConcMarkSweepGC
>> -J-XX:+UseTLAB -J-XX:NewSize=128m -J-XX:MaxNewSize=128m
>> -J-XX:MaxTenuringThreshold=0 -J-XX:SurvivorRatio=1024
>> -J-XX:+UseCMSInitiatingOccupancyOnly -J-XX:CMSInitiatingOccupancyFr
>> action=10
>> -J-XX:MaxGCPauseMillis=1000 -J-XX:InitiatingHeapOccupancyPercent=10
>> -J-XX:+UseCompressedOops -J-XX:ParallelGCThreads=8 -J-XX:ConcGCThreads=8
>> -J-XX:+PrintGCDetails -J-XX:+PrintGCTimeStamps -J-XX:+PrintGCDateStamps
>> -J-XX:+UseGCLogFileRotation -J-XX:NumberOfGCLogFiles=10
>> -J-XX:GCLogFileSize=100M -J-Xloggc:/tmp/ignite-gc.log
>> -J-Dcom.sun.management.jmxremote.port=49130
>>
>> Thanks for your help,
>> Charly
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/Small-value-for-IGNITE-ATOMIC-CACHE-DELETE
>> -HISTORY-SIZE-tp14037.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
>
> --
> Best regards,
> Ilya
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/Small-
> value-for-IGNITE-ATOMIC-CACHE-DELETE-HISTORY-SIZE-tp14037p14182.html
> To unsubscribe from Small value for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE,
> click here.
> NAML
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> ------------------------------
> View this message in context: Re: Small value for
> IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE
> <http://apache-ignite-users.70518.x6.nabble.com/Small-value-for-IGNITE-ATOMIC-CACHE-DELETE-HISTORY-SIZE-tp14037p14183.html>
>
> Sent from the Apache Ignite Users mailing list archive
> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
>



-- 
Best regards,
Ilya

Re: Small value for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE

Posted by charly <cd...@appnexus.com>.

It helps a lot. Thanks for the detailed example. I have a last question. Would it still be a potential issue to lower the queue size if we don't have any backup? Like if a new node pre-loads from the primary, any new changes will also be populated to the soon-to-be new primary?

Sent from my iPhone

On Jun 30, 2017, at 11:27 AM, Ilya Lantukh [via Apache Ignite Users] <ml...@n6.nabble.com>> wrote:

Hi Charly,

There is no special thread in charge of clearing the queue. Entry is cleared when you try to add new entry to the queue, but it is already full. So, once queue becomes full, it will always stay full, and lowering queue size obviously reduces heap consumption. If cache is destroyed, the queue should be cleared immediately.

However, if queue size is too low, it may lead to data inconsistency on unstable topology.

Here is an example:
Node A just joined the cluster, became primary node for some partitions and started preloading from node B, which is backup. Node B sends value V for key K, but before it was received and processed on node A user executed cache.remove(K).
- If queue size is 0, node A now doesn't have any information that entry was just removed and has to store (K, V) pair from node B. On the other hand, node B will remove K and now primary and backup nodes have different data.
- If queue size is N > 0, node A will save a tombstone entry for K. It will have all necessary information to understand that (K, V) pair from node B is older than current value and don't store it. Tombstone will be cleared after N other keys are removed.

Hope this helps.

On Thu, Jun 22, 2017 at 2:11 AM, charly <[hidden email]</user/SendEmail.jtp?type=node&node=14182&i=0>> wrote:
Hey everyone,

Thanks for your help,
Charly

--
Best regards,
Ilya

________________________________
If you reply to this email, your message will be added to the discussion below:
http://apache-ignite-users.70518.x6.nabble.com/Small-value-for-IGNITE-ATOMIC-CACHE-DELETE-HISTORY-SIZE-tp14037p14182.html
To unsubscribe from Small value for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE, click here<http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=14037&code=Y2Rlc3NhcnRAYXBwbmV4dXMuY29tfDE0MDM3fDE3Mzc3NzQ4OTA=>.
NAML<http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Small-value-for-IGNITE-ATOMIC-CACHE-DELETE-HISTORY-SIZE-tp14037p14183.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Small value for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE

Posted by Ilya Lantukh <il...@gridgain.com>.

Hi Charly,

There is no special thread in charge of clearing the queue. Entry is
cleared when you try to add new entry to the queue, but it is already full.
So, once queue becomes full, it will always stay full, and lowering queue
size obviously reduces heap consumption. If cache is destroyed, the queue
should be cleared immediately.

However, if queue size is too low, it may lead to data inconsistency on
unstable topology.

Here is an example:
Node A just joined the cluster, became primary node for some partitions and
started preloading from node B, which is backup. Node B sends value V for
key K, but before it was received and processed on node A user executed
cache.remove(K).
- If queue size is 0, node A now doesn't have any information that entry
was just removed and has to store (K, V) pair from node B. On the other
hand, node B will remove K and now primary and backup nodes have different
data.
- If queue size is N > 0, node A will save a tombstone entry for K. It will
have all necessary information to understand that (K, V) pair from node B
is older than current value and don't store it. Tombstone will be cleared
after N other keys are removed.

Hope this helps.


On Thu, Jun 22, 2017 at 2:11 AM, charly <cd...@appnexus.com> wrote:

> Hey everyone,
>
> We use Ignite 1.8 and see a difference in heap used when lowering the value
> for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE. I'm not sure to understand
> what
> it's going to happen if we decrease that value on production. Could someone
> clear that up for me please? From what I understand, deleted entries go in
> that queue for cleaning, where one thread per cache is in charge of
> cleaning
> the queue. But what if the queue is full? I might have missed that
> information in the documentation, my bad if I did.
>
> Maybe on the same subject, is it going to slow down the free up of the heap
> after cache.destroy()? We have a use-case where we delete 4 caches at the
> same time containing ~500k entries heavily indexed. When doing so, it takes
> few minutes for the heap to decrease as you can see here
> http://imgur.com/a/kPsSc . We would not want to extend the time to free up
> memory even more.
>
> More information about our setup:
>  - on heap (we'll eventually move to off heap but back then we could not
> make Ignite 1.8 to free up memory at all with our version of glibc. It
> would
> work with jmalloc but that was not permitted on production at that time).
>  - no backup
>  - atomicity mode: atomic
>  - cache mode: partitioned
>  - cluster of 4 nodes
>  - jvm options: -J-DIGNITE_LONG_OPERATIONS_DUMP_TIMEOUT=300000 -J-Xms12g
> -J-Xmx12g -J-server -J-XX:+UseParNewGC -J-XX:+UseConcMarkSweepGC
> -J-XX:+UseTLAB -J-XX:NewSize=128m -J-XX:MaxNewSize=128m
> -J-XX:MaxTenuringThreshold=0 -J-XX:SurvivorRatio=1024
> -J-XX:+UseCMSInitiatingOccupancyOnly -J-XX:CMSInitiatingOccupancyFraction
> =10
> -J-XX:MaxGCPauseMillis=1000 -J-XX:InitiatingHeapOccupancyPercent=10
> -J-XX:+UseCompressedOops -J-XX:ParallelGCThreads=8 -J-XX:ConcGCThreads=8
> -J-XX:+PrintGCDetails -J-XX:+PrintGCTimeStamps -J-XX:+PrintGCDateStamps
> -J-XX:+UseGCLogFileRotation -J-XX:NumberOfGCLogFiles=10
> -J-XX:GCLogFileSize=100M -J-Xloggc:/tmp/ignite-gc.log
> -J-Dcom.sun.management.jmxremote.port=49130
>
> Thanks for your help,
> Charly
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Small-value-for-IGNITE-ATOMIC-CACHE-
> DELETE-HISTORY-SIZE-tp14037.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Best regards,
Ilya