You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2020/01/13 14:28:26 UTC
[GitHub] [ignite] alex-plekhanov opened a new pull request #7245:
IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
alex-plekhanov opened a new pull request #7245: IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
URL: https://github.com/apache/ignite/pull/7245
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] zstan commented on a change in pull request #7245:
IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
Posted by GitBox <gi...@apache.org>.
zstan commented on a change in pull request #7245: IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
URL: https://github.com/apache/ignite/pull/7245#discussion_r366811227
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java
##########
@@ -118,6 +120,19 @@
* Used when persistence enabled.
*/
public class GridCacheOffheapManager extends IgniteCacheOffheapManagerImpl implements DbCheckpointListener {
+ /**
+ * Threshold to calculate limit for pages list on-heap caches.
+ * <p>
+ * Note: When a checkpoint is triggered, we need some amount of page memory to store pages list on-heap cache.
+ * If a checkpoint is triggered by "too many dirty pages" reason and pages list cache is rather big, we can get
+ * {@code IgniteOutOfMemoryException}. To prevent this, we can limit the total amount of cached page list buckets,
+ * assuming that checkpoint will be triggered if no more then 3/4 of pages will be marked as dirty (there will be
+ * at least 1/4 of clean pages) and each cached page list bucket can be stored to up to 2 pages (this value is not
+ * static, but depends on PagesCache.MAX_SIZE, so if PagesCache.MAX_SIZE > PagesListNodeIO#getCapacity it can take
+ * more than 2 pages). Also some amount of page memory needed to store page list metadata.
+ */
+ public static final double PAGE_LIST_CACHE_LIMIT_THRESHOLD = 0.1;
Review comment:
i still have no clue how u define this constant is it really enough to be static (not dynamically) calculated here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] zstan commented on a change in pull request #7245:
IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
Posted by GitBox <gi...@apache.org>.
zstan commented on a change in pull request #7245: IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
URL: https://github.com/apache/ignite/pull/7245#discussion_r366825189
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java
##########
@@ -118,6 +120,19 @@
* Used when persistence enabled.
*/
public class GridCacheOffheapManager extends IgniteCacheOffheapManagerImpl implements DbCheckpointListener {
+ /**
+ * Threshold to calculate limit for pages list on-heap caches.
+ * <p>
+ * Note: When a checkpoint is triggered, we need some amount of page memory to store pages list on-heap cache.
+ * If a checkpoint is triggered by "too many dirty pages" reason and pages list cache is rather big, we can get
+ * {@code IgniteOutOfMemoryException}. To prevent this, we can limit the total amount of cached page list buckets,
+ * assuming that checkpoint will be triggered if no more then 3/4 of pages will be marked as dirty (there will be
+ * at least 1/4 of clean pages) and each cached page list bucket can be stored to up to 2 pages (this value is not
+ * static, but depends on PagesCache.MAX_SIZE, so if PagesCache.MAX_SIZE > PagesListNodeIO#getCapacity it can take
+ * more than 2 pages). Also some amount of page memory needed to store page list metadata.
+ */
+ public static final double PAGE_LIST_CACHE_LIMIT_THRESHOLD = 0.1;
Review comment:
it really enough to be static (not dynamically) calculated here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] zstan commented on a change in pull request #7245:
IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
Posted by GitBox <gi...@apache.org>.
zstan commented on a change in pull request #7245: IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
URL: https://github.com/apache/ignite/pull/7245#discussion_r366811227
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java
##########
@@ -118,6 +120,19 @@
* Used when persistence enabled.
*/
public class GridCacheOffheapManager extends IgniteCacheOffheapManagerImpl implements DbCheckpointListener {
+ /**
+ * Threshold to calculate limit for pages list on-heap caches.
+ * <p>
+ * Note: When a checkpoint is triggered, we need some amount of page memory to store pages list on-heap cache.
+ * If a checkpoint is triggered by "too many dirty pages" reason and pages list cache is rather big, we can get
+ * {@code IgniteOutOfMemoryException}. To prevent this, we can limit the total amount of cached page list buckets,
+ * assuming that checkpoint will be triggered if no more then 3/4 of pages will be marked as dirty (there will be
+ * at least 1/4 of clean pages) and each cached page list bucket can be stored to up to 2 pages (this value is not
+ * static, but depends on PagesCache.MAX_SIZE, so if PagesCache.MAX_SIZE > PagesListNodeIO#getCapacity it can take
+ * more than 2 pages). Also some amount of page memory needed to store page list metadata.
+ */
+ public static final double PAGE_LIST_CACHE_LIMIT_THRESHOLD = 0.1;
Review comment:
i still have no clue how u define this constant is it really enough to be static (not dynamically) calculated here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] alex-plekhanov commented on a change in pull request
#7245: IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on
checkpoint
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #7245: IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
URL: https://github.com/apache/ignite/pull/7245#discussion_r366812927
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java
##########
@@ -118,6 +120,19 @@
* Used when persistence enabled.
*/
public class GridCacheOffheapManager extends IgniteCacheOffheapManagerImpl implements DbCheckpointListener {
+ /**
+ * Threshold to calculate limit for pages list on-heap caches.
+ * <p>
+ * Note: When a checkpoint is triggered, we need some amount of page memory to store pages list on-heap cache.
+ * If a checkpoint is triggered by "too many dirty pages" reason and pages list cache is rather big, we can get
+ * {@code IgniteOutOfMemoryException}. To prevent this, we can limit the total amount of cached page list buckets,
+ * assuming that checkpoint will be triggered if no more then 3/4 of pages will be marked as dirty (there will be
+ * at least 1/4 of clean pages) and each cached page list bucket can be stored to up to 2 pages (this value is not
+ * static, but depends on PagesCache.MAX_SIZE, so if PagesCache.MAX_SIZE > PagesListNodeIO#getCapacity it can take
+ * more than 2 pages). Also some amount of page memory needed to store page list metadata.
+ */
+ public static final double PAGE_LIST_CACHE_LIMIT_THRESHOLD = 0.1;
Review comment:
1/4 (clean pages) / 2 (at most 2 pages per bucket cache) + some reserve for metadata.
It's hard to calculate it dynamically, it depends on metadata size (which we don't know at runtime)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] alex-plekhanov commented on a change in pull request
#7245: IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on
checkpoint
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #7245: IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
URL: https://github.com/apache/ignite/pull/7245#discussion_r366831979
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java
##########
@@ -118,6 +120,19 @@
* Used when persistence enabled.
*/
public class GridCacheOffheapManager extends IgniteCacheOffheapManagerImpl implements DbCheckpointListener {
+ /**
+ * Threshold to calculate limit for pages list on-heap caches.
+ * <p>
+ * Note: When a checkpoint is triggered, we need some amount of page memory to store pages list on-heap cache.
+ * If a checkpoint is triggered by "too many dirty pages" reason and pages list cache is rather big, we can get
+ * {@code IgniteOutOfMemoryException}. To prevent this, we can limit the total amount of cached page list buckets,
+ * assuming that checkpoint will be triggered if no more then 3/4 of pages will be marked as dirty (there will be
+ * at least 1/4 of clean pages) and each cached page list bucket can be stored to up to 2 pages (this value is not
+ * static, but depends on PagesCache.MAX_SIZE, so if PagesCache.MAX_SIZE > PagesListNodeIO#getCapacity it can take
+ * more than 2 pages). Also some amount of page memory needed to store page list metadata.
+ */
+ public static final double PAGE_LIST_CACHE_LIMIT_THRESHOLD = 0.1;
Review comment:
We can't calculate it dynamically, so yes, reasonable static value is the best we can do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] zstan commented on a change in pull request #7245:
IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
Posted by GitBox <gi...@apache.org>.
zstan commented on a change in pull request #7245: IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
URL: https://github.com/apache/ignite/pull/7245#discussion_r366811267
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java
##########
@@ -118,6 +120,19 @@
* Used when persistence enabled.
*/
public class GridCacheOffheapManager extends IgniteCacheOffheapManagerImpl implements DbCheckpointListener {
+ /**
+ * Threshold to calculate limit for pages list on-heap caches.
+ * <p>
+ * Note: When a checkpoint is triggered, we need some amount of page memory to store pages list on-heap cache.
+ * If a checkpoint is triggered by "too many dirty pages" reason and pages list cache is rather big, we can get
+ * {@code IgniteOutOfMemoryException}. To prevent this, we can limit the total amount of cached page list buckets,
+ * assuming that checkpoint will be triggered if no more then 3/4 of pages will be marked as dirty (there will be
+ * at least 1/4 of clean pages) and each cached page list bucket can be stored to up to 2 pages (this value is not
+ * static, but depends on PagesCache.MAX_SIZE, so if PagesCache.MAX_SIZE > PagesListNodeIO#getCapacity it can take
+ * more than 2 pages). Also some amount of page memory needed to store page list metadata.
+ */
+ public static final double PAGE_LIST_CACHE_LIMIT_THRESHOLD = 0.1;
Review comment:
i still have no clue how u define this constant is it really enough to be static (not dynamically) calculated here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] zstan commented on a change in pull request #7245:
IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
Posted by GitBox <gi...@apache.org>.
zstan commented on a change in pull request #7245: IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
URL: https://github.com/apache/ignite/pull/7245#discussion_r366804536
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/freelist/PagesList.java
##########
@@ -1951,14 +1951,19 @@ public int cachedPagesCount(int bucket) {
/** Count of flush calls with empty cache. */
private int emptyFlushCnt;
+ /** Global (per data region) limit of caches for page lists. */
+ private final AtomicLong pagesCacheLimit;
+
/**
* Default constructor.
*/
- public PagesCache() {
+ public PagesCache(AtomicLong pagesCacheLimit) {
Review comment:
append @Nullable here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] asfgit closed pull request #7245: IGNITE-12530 Pages list
cache limit added to prevent IgniteOOME on checkpoint
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #7245: IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
URL: https://github.com/apache/ignite/pull/7245
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] zstan commented on a change in pull request #7245:
IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
Posted by GitBox <gi...@apache.org>.
zstan commented on a change in pull request #7245: IGNITE-12530 Pages list cache limit added to prevent IgniteOOME on checkpoint
URL: https://github.com/apache/ignite/pull/7245#discussion_r366178385
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java
##########
@@ -118,6 +120,17 @@
* Used when persistence enabled.
*/
public class GridCacheOffheapManager extends IgniteCacheOffheapManagerImpl implements DbCheckpointListener {
+ /**
+ * Threshold to calculate limit for pages list on-heap caches.
+ * <p>
+ * Note: When a checkpoint is triggered, we need some amount of page memory to store pages list on-heap cache.
+ * If a checkpoint is triggered by "too many dirty pages" reason and pages list cache is rather big, we can get
+ * {@code IgniteOutOfMemoryException}. To prevent this, we can limit the total amount of cached page list buckets,
+ * assuming that checkpoint will be triggered if no more then 3/4 of pages will be marked as dirty (there will be
+ * at least 1/4 of clean pages) and each cached page list bucket can be stored to up to 2 pages.
+ */
+ private static final double PAGE_LIST_CACHE_LIMIT_THRESHOLD = 1.0 / 4.0 / 2.0;
Review comment:
more exactly this formula is not static, but depends on PagesCache.MAX_SIZE, so if PagesCache.MAX_SIZE > (empty)PagesListNodeIO#getCapacity it can take more than 2 pages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services