You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "krkumar24061975@gmail.com" <kr...@gmail.com> on 2020/04/09 10:18:04 UTC
Checkpoint buffer error
Hi Guys - Occasionally my streamer throws exception saying DataStreamer
closed and when I dig into it, I found this error in the ignite logs "Failed
to allocate temporary buffer for checkpoint". What does this mean??
Thanx and Regards,
KR Kumar
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Checkpoint buffer error
Posted by Evgenii Zhuravlev <e....@gmail.com>.
Can you provide full logs from node?
пт, 17 апр. 2020 г. в 03:57, krkumar24061975@gmail.com <
krkumar24061975@gmail.com>:
> Its 2.7.6
>
> and configuration is as follows:
>
> <bean id="igniteContainer"
> class="com.pointillist.grid.ignite.IgniteSpringBean"
> depends-on="dataSource">
> <property name="configuration">
> <bean id="ignite.cfg"
> class="org.apache.ignite.configuration.IgniteConfiguration">
> <property name="workDirectory" value="${
> work.space}"></property>
> <property name="communicationSpi">
> <bean
> class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
> <property
> name="connectTimeout" value="10000"></property>
> <property
> name="socketWriteTimeout" value="30000"></property>
> </bean>
> </property>
> <property name="failureDetectionTimeout"
> value="120000"></property>
> <property name="rebalanceThreadPoolSize"
> value="8"></property>
> <property name="publicThreadPoolSize"
> value="64" />
> <property name="systemThreadPoolSize"
> value="32" />
> <property name="dataStorageConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataStorageConfiguration">
> <property
> name="writeThrottlingEnabled" value="true"></property>
> <property
> name="defaultDataRegionConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property
> name="initialSize" value="#{512L * 1024 *
> 1024}"></property>
> <property
> name="maxSize" value="#{20L * 1024 * 1024 * 1024}" />
> <property
> name="persistenceEnabled" value="true" />
> </bean>
> </property>
> <property
> name="dataRegionConfigurations">
> <list>
> <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
>
> <property name="name" value="indexMainDataRegion" />
>
> <property name="initialSize" value="#{512L * 1024 * 1024}" />
>
> <property name="maxSize" value="#{20L * 1024 * 1024 * 1024}" />
>
> <property name="persistenceEnabled" value="true" />
> </bean>
> <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
>
> <property name="name" value="indexDtlDataRegion" />
>
> <property name="initialSize" value="#{512L * 1024 * 1024}" />
>
> <property name="maxSize" value="#{20L * 1024 * 1024 * 1024}" />
>
> <property name="persistenceEnabled" value="true" />
> </bean>
> </list>
> </property>
> <property name="pageSize"
> value="#{4 * 1024}" />
> <property
> name="storagePath" value="${grid.data}" />
> <property name="walPath"
> value="${grid.wal}" />
> <property
> name="walArchivePath" value="${grid.wal}" />
> <property name="walMode"
> value="BACKGROUND" />
> <property
> name="walFlushFrequency" value="10000"></property>
> </bean>
> </property>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Checkpoint buffer error
Posted by "krkumar24061975@gmail.com" <kr...@gmail.com>.
Its 2.7.6
and configuration is as follows:
<bean id="igniteContainer"
class="com.pointillist.grid.ignite.IgniteSpringBean"
depends-on="dataSource">
<property name="configuration">
<bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="workDirectory" value="${work.space}"></property>
<property name="communicationSpi">
<bean
class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="connectTimeout" value="10000"></property>
<property name="socketWriteTimeout" value="30000"></property>
</bean>
</property>
<property name="failureDetectionTimeout" value="120000"></property>
<property name="rebalanceThreadPoolSize" value="8"></property>
<property name="publicThreadPoolSize" value="64" />
<property name="systemThreadPoolSize" value="32" />
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="writeThrottlingEnabled" value="true"></property>
<property name="defaultDataRegionConfiguration">
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="initialSize" value="#{512L * 1024 *
1024}"></property>
<property name="maxSize" value="#{20L * 1024 * 1024 * 1024}" />
<property name="persistenceEnabled" value="true" />
</bean>
</property>
<property name="dataRegionConfigurations">
<list>
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="indexMainDataRegion" />
<property name="initialSize" value="#{512L * 1024 * 1024}" />
<property name="maxSize" value="#{20L * 1024 * 1024 * 1024}" />
<property name="persistenceEnabled" value="true" />
</bean>
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="indexDtlDataRegion" />
<property name="initialSize" value="#{512L * 1024 * 1024}" />
<property name="maxSize" value="#{20L * 1024 * 1024 * 1024}" />
<property name="persistenceEnabled" value="true" />
</bean>
</list>
</property>
<property name="pageSize" value="#{4 * 1024}" />
<property name="storagePath" value="${grid.data}" />
<property name="walPath" value="${grid.wal}" />
<property name="walArchivePath" value="${grid.wal}" />
<property name="walMode" value="BACKGROUND" />
<property name="walFlushFrequency" value="10000"></property>
</bean>
</property>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Checkpoint buffer error
Posted by "krkumar24061975@gmail.com" <kr...@gmail.com>.
Its 2.7.6
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Checkpoint buffer error
Posted by Evgenii Zhuravlev <e....@gmail.com>.
Hi,
What version of Ignite do you use?
Evgenii
ср, 15 апр. 2020 г. в 19:26, krkumar24061975@gmail.com <
krkumar24061975@gmail.com>:
> Hi - Here is the complete stack trace from the logs
>
> [2020-04-15 13:43:07,271][ERROR][data-streamer-stripe-2-#51][root] Failed
> to
> set initial value for cache entry: DataStreamerEntry
> [key=UserKeyCacheObjectImpl [part=207, val=50792583101, hasValBytes=true],
> val=CacheObjectByteArrayImpl [arrLen=403]]
> class
>
> org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
> Runtime failure on search row: SearchRow [key=KeyCacheObjectImpl [part=207,
> val=50792583101, hasValBytes=true], hash=-747024458, cacheId=0]
> at
>
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1815)
> at
>
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1642)
> at
>
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1625)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1935)
> at
>
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:428)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:4248)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:3391)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheEntryEx.initialValue(GridCacheEntryEx.java:766)
> at
>
> org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$IsolatedUpdater.receive(DataStreamerImpl.java:2265)
> at
>
> org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
> at
>
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6820)
> at
>
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:967)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteException: Failed to allocate
> temporary buffer for checkpoint (increase checkpointPageBufferSize
> configuration property): default
> at
>
> org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.postWriteLockPage(PageMemoryImpl.java:1575)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.tryWriteLockPage(PageMemoryImpl.java:1546)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.tryWriteLock(PageMemoryImpl.java:479)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writeLock(PageHandler.java:400)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.DataStructure.tryWriteLock(DataStructure.java:161)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.writeLockPage(PagesList.java:1021)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.put(PagesList.java:641)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList$WriteRowHandler.run(AbstractFreeList.java:164)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList$WriteRowHandler.run(AbstractFreeList.java:136)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:279)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:296)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.insertDataRow(AbstractFreeList.java:500)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.freelist.CacheFreeListImpl.insertDataRow(CacheFreeListImpl.java:59)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.freelist.CacheFreeListImpl.insertDataRow(CacheFreeListImpl.java:35)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.RowStore.addRow(RowStore.java:103)
> at
>
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.createRow(IgniteCacheOffheapManagerImpl.java:1695)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.createRow(GridCacheOffheapManager.java:1910)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:5701)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:5643)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:3722)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.access$5900(BPlusTree.java:3616)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1898)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1875)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1875)
> at
>
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1782)
> ... 15 more
> [2020-04-15 13:43:07,270][ERROR][data-streamer-stripe-47-#96][root] Failed
> to set initial value for cache entry: DataStreamerEntry
> [key=UserKeyCacheObjectImpl [part=222, val=37165764502, hasValBytes=true],
> val=CacheObjectByteArrayImpl [arrLen=524]]
> class
>
> org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
> Runtime failure on search row: SearchRow [key=KeyCacheObjectImpl [part=222,
> val=37165764502, hasValBytes=true], hash=-1488941154, cacheId=0]
> at
>
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1815)
> at
>
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1642)
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Checkpoint buffer error
Posted by "krkumar24061975@gmail.com" <kr...@gmail.com>.
Hi - Here is the complete stack trace from the logs
[2020-04-15 13:43:07,271][ERROR][data-streamer-stripe-2-#51][root] Failed to
set initial value for cache entry: DataStreamerEntry
[key=UserKeyCacheObjectImpl [part=207, val=50792583101, hasValBytes=true],
val=CacheObjectByteArrayImpl [arrLen=403]]
class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on search row: SearchRow [key=KeyCacheObjectImpl [part=207,
val=50792583101, hasValBytes=true], hash=-747024458, cacheId=0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1815)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1642)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1625)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1935)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:428)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:4248)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:3391)
at
org.apache.ignite.internal.processors.cache.GridCacheEntryEx.initialValue(GridCacheEntryEx.java:766)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$IsolatedUpdater.receive(DataStreamerImpl.java:2265)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6820)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:967)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteException: Failed to allocate
temporary buffer for checkpoint (increase checkpointPageBufferSize
configuration property): default
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.postWriteLockPage(PageMemoryImpl.java:1575)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.tryWriteLockPage(PageMemoryImpl.java:1546)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.tryWriteLock(PageMemoryImpl.java:479)
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writeLock(PageHandler.java:400)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.tryWriteLock(DataStructure.java:161)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.writeLockPage(PagesList.java:1021)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.put(PagesList.java:641)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList$WriteRowHandler.run(AbstractFreeList.java:164)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList$WriteRowHandler.run(AbstractFreeList.java:136)
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:279)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:296)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.insertDataRow(AbstractFreeList.java:500)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.CacheFreeListImpl.insertDataRow(CacheFreeListImpl.java:59)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.CacheFreeListImpl.insertDataRow(CacheFreeListImpl.java:35)
at
org.apache.ignite.internal.processors.cache.persistence.RowStore.addRow(RowStore.java:103)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.createRow(IgniteCacheOffheapManagerImpl.java:1695)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.createRow(GridCacheOffheapManager.java:1910)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:5701)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:5643)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:3722)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.access$5900(BPlusTree.java:3616)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1898)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1875)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1875)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1782)
... 15 more
[2020-04-15 13:43:07,270][ERROR][data-streamer-stripe-47-#96][root] Failed
to set initial value for cache entry: DataStreamerEntry
[key=UserKeyCacheObjectImpl [part=222, val=37165764502, hasValBytes=true],
val=CacheObjectByteArrayImpl [arrLen=524]]
class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on search row: SearchRow [key=KeyCacheObjectImpl [part=222,
val=37165764502, hasValBytes=true], hash=-1488941154, cacheId=0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1815)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1642)
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Checkpoint buffer error
Posted by Evgenii Zhuravlev <e....@gmail.com>.
Hi,
Can you attach log files from all nodes?
Evgenii
чт, 9 апр. 2020 г. в 03:18, krkumar24061975@gmail.com <
krkumar24061975@gmail.com>:
> Hi Guys - Occasionally my streamer throws exception saying DataStreamer
> closed and when I dig into it, I found this error in the ignite logs
> "Failed
> to allocate temporary buffer for checkpoint". What does this mean??
>
>
>
> Thanx and Regards,
> KR Kumar
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>