[jira] [Created] (IGNITE-8790) JVM crash during memory recovery

["Alexey Goncharuk (JIRA)" <ji...@apache.org>]

Alexey Goncharuk created IGNITE-8790:
----------------------------------------

             Summary: JVM crash during memory recovery
                 Key: IGNITE-8790
                 URL: https://issues.apache.org/jira/browse/IGNITE-8790
             Project: Ignite
          Issue Type: Bug
            Reporter: Alexey Goncharuk


I've observed the following JVM crash after one of the Ignite node restarts on 2.5 (only relevant part is kept):

{code}
Stack: [0x00007f16f40b8000,0x00007f16f41b9000],  sp=0x00007f16f41b7308,  free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x803675]
J 868  sun.misc.Unsafe.copyMemory(Ljava/lang/Object;JLjava/lang/Object;JJ)V (0 bytes) @ 0x00007f173d351ca1 [0x00007f173d351bc0+0xe1]
J 3023 C1 org.apache.ignite.internal.util.GridUnsafe.copyMemory(Ljava/lang/Object;JLjava/lang/Object;JJ)V (77 bytes) @ 0x00007f173d9e8d64 [0x00007f173d9e8ae0+0x284]
J 2991 C1 org.apache.ignite.internal.pagemem.PageUtils.putBytes(JI[B)V (73 bytes) @ 0x00007f173d9e1dbc [0x00007f173d9e1d00+0xbc]
j  org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(Lorg/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager$CheckpointStatus;ZLorg/apache/ignite/internal/processors/cache/persistence/pagemem/PageMemoryEx;)Lorg/apache/ignite/internal/pagemem/wal/WALPointer;+568
j  org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(Lorg/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager$CheckpointStatus;)Lorg/apache/ignite/internal/pagemem/wal/WALPointer;+13
j  org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(Ljava/util/List;)V+173
j  org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(Z)Lorg/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionsExchangeFuture$ExchangeType;+311
j  org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(Z)V+574
j  org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0()V+547
j  org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body()V+3
j  org.apache.ignite.internal.util.worker.GridWorker.run()V+82
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
V  [libjvm.so+0x695b96]
V  [libjvm.so+0x6960a1]
V  [libjvm.so+0x696537]
V  [libjvm.so+0x71596e]
V  [libjvm.so+0xa7f243]
V  [libjvm.so+0xa7f38c]
V  [libjvm.so+0x92e0f8]
C  [libpthread.so.0+0x76ba]  start_thread+0xca
{code}

Looks like that the issue is caused by a page which ID was rotated and the node failed before checkpoint is finished. Then, on the second node restart, the page was written to the disk, but node was stopped again before the checkpoint marker was written.
Then, on second node restart we attempt to write-lock the page, but lock fails because the page tag logged to WAL is different then the one written in the store.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)