You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Denis Chudov (Jira)" <ji...@apache.org> on 2022/05/20 12:22:00 UTC
[jira] [Commented] (IGNITE-16922) Getting an entry with expiry policy causes IgniteOutOfMemoryException
[ https://issues.apache.org/jira/browse/IGNITE-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540101#comment-17540101 ]
Denis Chudov commented on IGNITE-16922:
---------------------------------------
This happens because Ignite stores TTL in row itself and needs to update row, and in-place update doesn't happen because of the condition in {{IgniteCacheOffheapManagerImpl.CacheDataStoreImpl#createRow}}:
{code:java}
if (canUpdateOldRow(cctx, oldRow, dataRow) && rowStore.updateRow(oldRow.link(), dataRow, grp.statisticsHolderData()))
{code}
which return false and makes Ignite add new row first, and then remove old one.
> Getting an entry with expiry policy causes IgniteOutOfMemoryException
> ---------------------------------------------------------------------
>
> Key: IGNITE-16922
> URL: https://issues.apache.org/jira/browse/IGNITE-16922
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 2.13
> Reporter: Alexey Kukushkin
> Priority: Major
> Labels: cggg
> Original Estimate: 64h
> Remaining Estimate: 64h
>
> {{IgniteCache#get(key)}} operation causes {{IgniteOutOfMemoryException}} if {{AccessedExpiryPolicy}} or {{TouchedExpiryPolicy}} is enabled for the {{key}} and Ignite has not enough storage for another entry of the same or bigger size.
> This happens because:
> # Ignite needs to update TTL
> # TTL is part of the entry and Ignite overwrites full entry to update the TTL
> # The problem is Ignite runs common code that checks if Ignite has enough storage to write the entry with updated TTL back. The check fails causing the {{IgniteCache#get(key)}} operation to throw {{IgniteOutOfMemoryException}}.
> # This behavior is very confusing for Ignite users: why would a "read" operation throw Ignite OOM?
> Can we update the TTL atomically and skip the storage size check?
> Please enhance Ignite not to throw Ignite OOM on {{get}}.
> Stack trace:
> {code:java}
> [2022-05-20 15:08:20,025][ERROR][sys-stripe-6-#8%ignite.IgniteOOMOnGet0%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.mem.IgniteOutOfMemoryException: Out of memory in data region [name=default, initSize=18.1 MiB, maxSize=18.1 MiB, persistenceEnabled=false] Try the following:
> ^-- Increase maximum off-heap memory size (DataRegionConfiguration.maxSize)
> ^-- Enable Ignite persistence (DataRegionConfiguration.persistenceEnabled)
> ^-- Enable eviction or expiration policies]]
> class org.apache.ignite.internal.mem.IgniteOutOfMemoryException: Out of memory in data region [name=default, initSize=18.1 MiB, maxSize=18.1 MiB, persistenceEnabled=false] Try the following:
> ^-- Increase maximum off-heap memory size (DataRegionConfiguration.maxSize)
> ^-- Enable Ignite persistence (DataRegionConfiguration.persistenceEnabled)
> ^-- Enable eviction or expiration policies
> at org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager.ensureFreeSpaceForInsert(IgniteCacheDatabaseSharedManager.java:1234)
> at org.apache.ignite.internal.processors.cache.persistence.RowStore.addRow(RowStore.java:108)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.createRow(IgniteCacheOffheapManagerImpl.java:1962)
> at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:5767)
> at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:5695)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:4131)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:2121)
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1997)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1860)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1843)
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:471)
> at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:4164)
> at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:4140)
> at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.updateTtl(GridCacheMapEntry.java:2961)
> at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.updateTtl(GridCacheMapEntry.java:2934)
> at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGet0(GridCacheMapEntry.java:825)
> at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGetVersioned(GridCacheMapEntry.java:704)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getAllAsync0(GridDhtCacheAdapter.java:851)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtAllAsync(GridDhtCacheAdapter.java:691)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.getAsync(GridDhtGetSingleFuture.java:413)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map0(GridDhtGetSingleFuture.java:289)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map(GridDhtGetSingleFuture.java:270)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.init(GridDhtGetSingleFuture.java:186)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtSingleAsync(GridDhtCacheAdapter.java:1156)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetRequest(GridDhtCacheAdapter.java:1174)
> at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$200(GridDhtAtomicCache.java:151)
> at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$4.apply(GridDhtAtomicCache.java:278)
> at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$4.apply(GridDhtAtomicCache.java:273)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1150)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1727)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1334)
> at org.apache.ignite.internal.managers.communication.GridIoManager$8.execute(GridIoManager.java:1218)
> at org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:54)
> at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:567)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
> at java.base/java.lang.Thread.run(Thread.java:833)
> {code}
> Reproducer:
> {code:java}
> package org.apache.ignite;
> import java.util.concurrent.ThreadLocalRandom;
> import java.util.concurrent.TimeUnit;
> import java.util.concurrent.atomic.AtomicBoolean;
> import java.util.concurrent.atomic.AtomicInteger;
> import javax.cache.expiry.Duration;
> import javax.cache.expiry.TouchedExpiryPolicy;
> import org.apache.ignite.configuration.CacheConfiguration;
> import org.apache.ignite.configuration.DataRegionConfiguration;
> import org.apache.ignite.configuration.DataStorageConfiguration;
> import org.apache.ignite.configuration.IgniteConfiguration;
> import org.apache.ignite.internal.IgniteInternalFuture;
> import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
> import org.junit.Test;
> import static java.lang.Math.max;
> import static org.apache.ignite.testframework.GridTestUtils.runMultiThreadedAsync;
> public class IgniteOOMOnGet extends GridCommonAbstractTest {
> @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
> return super.getConfiguration(igniteInstanceName)
> .setDataStorageConfiguration(
> new DataStorageConfiguration().setDefaultDataRegionConfiguration(
> new DataRegionConfiguration().setMaxSize(19_000_000)
> )
> );
> }
> @Test
> public void test() throws Exception{
> Ignite ignite = startGrid(0);
> Ignite client = startClientGrid(1);
> IgniteCache<Integer, byte[]> cache = client.createCache(
> new CacheConfiguration<Integer, byte[]>(DEFAULT_CACHE_NAME)
> .setExpiryPolicyFactory(
> TouchedExpiryPolicy.factoryOf(new Duration(TimeUnit.SECONDS, 100_000))
> )
> );
> AtomicInteger maxKey = new AtomicInteger();
> for (int i = 0; i < 85; i++) {
> cache.put(i, new byte[10_000]);
> while (true) {
> int mk = maxKey.get();
> if (maxKey.compareAndSet(mk, max(mk, i)))
> break;
> }
> }
> AtomicBoolean enabled = new AtomicBoolean(true);
> IgniteInternalFuture fut = runMultiThreadedAsync(() -> {
> ThreadLocalRandom rnd = ThreadLocalRandom.current();
> while (enabled.get()) {
> cache.get(rnd.nextInt(maxKey.get()));
> }
> }, 8, "getterThread");
> doSleep(10_000);
> enabled.set(false);
> fut.get();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)