You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Vladislav Pyatkov (Jira)" <ji...@apache.org> on 2024/06/27 14:12:00 UTC
[jira] [Updated] (IGNITE-22598) Failed to allocate temporary buffer for checkpoint
[ https://issues.apache.org/jira/browse/IGNITE-22598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vladislav Pyatkov updated IGNITE-22598:
---------------------------------------
Description:
h3. Motivation
Many exception might appear in log of the thrutput test. After the partition storage is in undefined state. Notsurprised the te continue work with the storage leads to another issues.
{noformat}
2024-06-27 12:19:46:881 +0300 [INFO][%poc-tester-SERVER-192.168.1.41-id-0%JRaft-FSMCaller-Disruptor_stripe_6-0][ActionRequestProcessor] Error occurred on a user's state machine
org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to put value into index
at org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:123)
at org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busy(AbstractPageMemoryIndexStorage.java:336)
at org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busyNonDataRead(AbstractPageMemoryIndexStorage.java:317)
at org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.put(PageMemoryHashIndexStorage.java:109)
at org.apache.ignite.internal.table.distributed.TableSchemaAwareIndexStorage.put(TableSchemaAwareIndexStorage.java:83)
at org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.putToIndex(IndexUpdateHandler.java:270)
at org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.addToIndexes(IndexUpdateHandler.java:69)
at org.apache.ignite.internal.table.distributed.StorageUpdateHandler.tryProcessRow(StorageUpdateHandler.java:173)
at org.apache.ignite.internal.table.distributed.StorageUpdateHandler.lambda$handleUpdate$0(StorageUpdateHandler.java:114)
at org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.lambda$runConsistently$0(PersistentPageMemoryMvPartitionStorage.java:165)
at org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.busy(AbstractPageMemoryMvPartitionStorage.java:668)
at org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.runConsistently(PersistentPageMemoryMvPartitionStorage.java:155)
at org.apache.ignite.internal.table.distributed.raft.snapshot.outgoing.SnapshotAwarePartitionDataStorage.runConsistently(SnapshotAwarePartitionDataStorage.java:76)
at org.apache.ignite.internal.table.distributed.StorageUpdateHandler.handleUpdate(StorageUpdateHandler.java:109)
at org.apache.ignite.internal.table.distributed.raft.PartitionListener.handleUpdateCommand(PartitionListener.java:289)
at org.apache.ignite.internal.table.distributed.raft.PartitionListener.lambda$onWrite$1(PartitionListener.java:209)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at org.apache.ignite.internal.table.distributed.raft.PartitionListener.onWrite(PartitionListener.java:166)
at org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:702)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:571)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:539)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:458)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:131)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:125)
at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:326)
at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:283)
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.ignite.internal.pagememory.tree.CorruptedTreeException: IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d B+Tree is corrupted [groupId=10, pageIds=[562988608185341], groupName=10, msg=Runtime failure on search row: org.apache.ignite.internal.storage.pagememory.index.hash.HashIndexRow@60f9a709]
at org.apache.ignite.internal.pagememory.tree.BplusTree.corruptedTreeException(BplusTree.java:6660)
at org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2109)
at org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:119)
... 28 more
Caused by: org.apache.ignite.internal.lang.IgniteInternalException: IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to allocate temporary buffer for checkpoint (increase checkpointPageBufferSize configuration property): default
at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.postWriteLockPage(PersistentPageMemory.java:1099)
at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLockPage(PersistentPageMemory.java:1084)
at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:449)
at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:434)
at org.apache.ignite.internal.pagememory.util.PageHandler.writeLock(PageHandler.java:398)
at org.apache.ignite.internal.pagememory.util.PageHandler.writePage(PageHandler.java:347)
at org.apache.ignite.internal.pagememory.datastructure.DataStructure.write(DataStructure.java:319)
at org.apache.ignite.internal.pagememory.tree.BplusTree.access$5000(BplusTree.java:193)
at org.apache.ignite.internal.pagememory.tree.BplusTree$Put.tryInsert(BplusTree.java:4102)
at org.apache.ignite.internal.pagememory.tree.BplusTree$Invoke.onNotFound(BplusTree.java:4360)
at org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2186)
at org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161)
at org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161)
at org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2083)
... 29 more
{noformat}
h3. Definition of done
Exceptions like that is in the attached log have not to appear.
was:
h3. Motivation
Many exception might appear in log of the thrutput test. After the partition storage is in undefined state. Notsurprised the te continue work with the storage leads to another issues.
{noformat}
2024-06-27 12:19:46:881 +0300 [INFO][%poc-tester-SERVER-192.168.1.41-id-0%JRaft-FSMCaller-Disruptor_stripe_6-0][ActionRequestProcessor] Error occurred on a user's state machine
org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to put value into index
at org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:123)
at org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busy(AbstractPageMemoryIndexStorage.java:336)
at org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busyNonDataRead(AbstractPageMemoryIndexStorage.java:317)
at org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.put(PageMemoryHashIndexStorage.java:109)
at org.apache.ignite.internal.table.distributed.TableSchemaAwareIndexStorage.put(TableSchemaAwareIndexStorage.java:83)
at org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.putToIndex(IndexUpdateHandler.java:270)
at org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.addToIndexes(IndexUpdateHandler.java:69)
at org.apache.ignite.internal.table.distributed.StorageUpdateHandler.tryProcessRow(StorageUpdateHandler.java:173)
at org.apache.ignite.internal.table.distributed.StorageUpdateHandler.lambda$handleUpdate$0(StorageUpdateHandler.java:114)
at org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.lambda$runConsistently$0(PersistentPageMemoryMvPartitionStorage.java:165)
at org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.busy(AbstractPageMemoryMvPartitionStorage.java:668)
at org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.runConsistently(PersistentPageMemoryMvPartitionStorage.java:155)
at org.apache.ignite.internal.table.distributed.raft.snapshot.outgoing.SnapshotAwarePartitionDataStorage.runConsistently(SnapshotAwarePartitionDataStorage.java:76)
at org.apache.ignite.internal.table.distributed.StorageUpdateHandler.handleUpdate(StorageUpdateHandler.java:109)
at org.apache.ignite.internal.table.distributed.raft.PartitionListener.handleUpdateCommand(PartitionListener.java:289)
at org.apache.ignite.internal.table.distributed.raft.PartitionListener.lambda$onWrite$1(PartitionListener.java:209)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at org.apache.ignite.internal.table.distributed.raft.PartitionListener.onWrite(PartitionListener.java:166)
at org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:702)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:571)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:539)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:458)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:131)
at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:125)
at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:326)
at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:283)
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.ignite.internal.pagememory.tree.CorruptedTreeException: IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d B+Tree is corrupted [groupId=10, pageIds=[562988608185341], groupName=10, msg=Runtime failure on search row: org.apache.ignite.internal.storage.pagememory.index.hash.HashIndexRow@60f9a709]
at org.apache.ignite.internal.pagememory.tree.BplusTree.corruptedTreeException(BplusTree.java:6660)
at org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2109)
at org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:119)
... 28 more
Caused by: org.apache.ignite.internal.lang.IgniteInternalException: IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to allocate temporary buffer for checkpoint (increase checkpointPageBufferSize configuration property): default
at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.postWriteLockPage(PersistentPageMemory.java:1099)
at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLockPage(PersistentPageMemory.java:1084)
at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:449)
at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:434)
at org.apache.ignite.internal.pagememory.util.PageHandler.writeLock(PageHandler.java:398)
at org.apache.ignite.internal.pagememory.util.PageHandler.writePage(PageHandler.java:347)
at org.apache.ignite.internal.pagememory.datastructure.DataStructure.write(DataStructure.java:319)
at org.apache.ignite.internal.pagememory.tree.BplusTree.access$5000(BplusTree.java:193)
at org.apache.ignite.internal.pagememory.tree.BplusTree$Put.tryInsert(BplusTree.java:4102)
at org.apache.ignite.internal.pagememory.tree.BplusTree$Invoke.onNotFound(BplusTree.java:4360)
at org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2186)
at org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161)
at org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161)
at org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2083)
... 29 more
{noformat}
Definition of done
Exceptions like that is in the attached log have not to appear.
> Failed to allocate temporary buffer for checkpoint
> --------------------------------------------------
>
> Key: IGNITE-22598
> URL: https://issues.apache.org/jira/browse/IGNITE-22598
> Project: Ignite
> Issue Type: Bug
> Reporter: Vladislav Pyatkov
> Priority: Major
> Labels: ignite-3
> Attachments: poc-tester-SERVER-192.168.1.41-id-0-2024-06-27-09-14-17-client.log.2
>
>
> h3. Motivation
> Many exception might appear in log of the thrutput test. After the partition storage is in undefined state. Notsurprised the te continue work with the storage leads to another issues.
> {noformat}
> 2024-06-27 12:19:46:881 +0300 [INFO][%poc-tester-SERVER-192.168.1.41-id-0%JRaft-FSMCaller-Disruptor_stripe_6-0][ActionRequestProcessor] Error occurred on a user's state machine
> org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to put value into index
> at org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:123)
> at org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busy(AbstractPageMemoryIndexStorage.java:336)
> at org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busyNonDataRead(AbstractPageMemoryIndexStorage.java:317)
> at org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.put(PageMemoryHashIndexStorage.java:109)
> at org.apache.ignite.internal.table.distributed.TableSchemaAwareIndexStorage.put(TableSchemaAwareIndexStorage.java:83)
> at org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.putToIndex(IndexUpdateHandler.java:270)
> at org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.addToIndexes(IndexUpdateHandler.java:69)
> at org.apache.ignite.internal.table.distributed.StorageUpdateHandler.tryProcessRow(StorageUpdateHandler.java:173)
> at org.apache.ignite.internal.table.distributed.StorageUpdateHandler.lambda$handleUpdate$0(StorageUpdateHandler.java:114)
> at org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.lambda$runConsistently$0(PersistentPageMemoryMvPartitionStorage.java:165)
> at org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.busy(AbstractPageMemoryMvPartitionStorage.java:668)
> at org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.runConsistently(PersistentPageMemoryMvPartitionStorage.java:155)
> at org.apache.ignite.internal.table.distributed.raft.snapshot.outgoing.SnapshotAwarePartitionDataStorage.runConsistently(SnapshotAwarePartitionDataStorage.java:76)
> at org.apache.ignite.internal.table.distributed.StorageUpdateHandler.handleUpdate(StorageUpdateHandler.java:109)
> at org.apache.ignite.internal.table.distributed.raft.PartitionListener.handleUpdateCommand(PartitionListener.java:289)
> at org.apache.ignite.internal.table.distributed.raft.PartitionListener.lambda$onWrite$1(PartitionListener.java:209)
> at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
> at org.apache.ignite.internal.table.distributed.raft.PartitionListener.onWrite(PartitionListener.java:166)
> at org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:702)
> at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:571)
> at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:539)
> at org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:458)
> at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:131)
> at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:125)
> at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:326)
> at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:283)
> at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
> at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: org.apache.ignite.internal.pagememory.tree.CorruptedTreeException: IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d B+Tree is corrupted [groupId=10, pageIds=[562988608185341], groupName=10, msg=Runtime failure on search row: org.apache.ignite.internal.storage.pagememory.index.hash.HashIndexRow@60f9a709]
> at org.apache.ignite.internal.pagememory.tree.BplusTree.corruptedTreeException(BplusTree.java:6660)
> at org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2109)
> at org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:119)
> ... 28 more
> Caused by: org.apache.ignite.internal.lang.IgniteInternalException: IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to allocate temporary buffer for checkpoint (increase checkpointPageBufferSize configuration property): default
> at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.postWriteLockPage(PersistentPageMemory.java:1099)
> at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLockPage(PersistentPageMemory.java:1084)
> at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:449)
> at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:434)
> at org.apache.ignite.internal.pagememory.util.PageHandler.writeLock(PageHandler.java:398)
> at org.apache.ignite.internal.pagememory.util.PageHandler.writePage(PageHandler.java:347)
> at org.apache.ignite.internal.pagememory.datastructure.DataStructure.write(DataStructure.java:319)
> at org.apache.ignite.internal.pagememory.tree.BplusTree.access$5000(BplusTree.java:193)
> at org.apache.ignite.internal.pagememory.tree.BplusTree$Put.tryInsert(BplusTree.java:4102)
> at org.apache.ignite.internal.pagememory.tree.BplusTree$Invoke.onNotFound(BplusTree.java:4360)
> at org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2186)
> at org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161)
> at org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161)
> at org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2083)
> ... 29 more
> {noformat}
> h3. Definition of done
> Exceptions like that is in the attached log have not to appear.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)