You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Maxim Muzafarov (Jira)" <ji...@apache.org> on 2019/10/02 12:07:00 UTC
[jira] [Updated] (IGNITE-11217) After cluster kill and restart with
new jvm option - one of node fail with failure type - Failed to read WAL
record at position
[ https://issues.apache.org/jira/browse/IGNITE-11217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maxim Muzafarov updated IGNITE-11217:
-------------------------------------
Fix Version/s: 2.9
> After cluster kill and restart with new jvm option - one of node fail with failure type - Failed to read WAL record at position
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: IGNITE-11217
> URL: https://issues.apache.org/jira/browse/IGNITE-11217
> Project: Ignite
> Issue Type: Bug
> Components: data structures, mvcc
> Affects Versions: 2.7
> Reporter: ARomantsov
> Priority: Critical
> Labels: mvcc_stability
> Fix For: 2.8, 2.9
>
>
> Test scenario:
> - Start two server load
> - execute operations (put/put-all/streamer/remove)
> - execute client that collect checksum
> - kill nodes
> - add JVM options to server - DIGNITE_WAL_LOG_TX_RECORDS=true, DIGNITE_RECOVERY_VERBOSE_LOGGING
> - start nodes
> - activate them
> Found that one node fail
> {code:java}
> [23:16:06,230][INFO][main][GridCacheProcessor] Finished restoring partition state for local groups [groupsProcessed=45, partitionsProcessed=764, time=557ms]
> [23:16:06,280][INFO][main][FileWriteAheadLogManager] Stopping WAL iteration due to an exception: Failed to read WAL record at position: 27700852 size: 67108864, ptr=FileWALPointer [idx=0, fileOff=27700852, len=0]
> [23:16:06,298][SEVERE][main][IgniteKernal] Exception during start processors, node will be stopped and close connections
> class org.apache.ignite.IgniteCheckedException: Failed to read WAL record at position: 27700852 size: 67108864
> at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.handleRecordException(AbstractWalRecordsIterator.java:305)
> at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.handleRecordException(FileWriteAheadLogManager.java:2691)
> at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.advanceRecord(AbstractWalRecordsIterator.java:269)
> at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.advance(AbstractWalRecordsIterator.java:158)
> at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.onNext(AbstractWalRecordsIterator.java:127)
> at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.onNext(AbstractWalRecordsIterator.java:53)
> at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.nextX(GridCloseableIteratorAdapter.java:41)
> at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.tailPointer(GridCacheDatabaseSharedManager.java:1984)
> at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.startMemoryRestore(GridCacheDatabaseSharedManager.java:1958)
> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1050)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2040)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1732)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
> at org.apache.ignite.Ignition.start(Ignition.java:348)
> at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read WAL record at position: 27700852 size: 67108864
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV1Serializer.readWithCrc(RecordV1Serializer.java:395)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV2Serializer.readRecord(RecordV2Serializer.java:236)
> at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.advanceRecord(AbstractWalRecordsIterator.java:254)
> ... 17 more
> Caused by: java.io.IOException: class org.apache.ignite.IgniteCheckedException: Unsupported protocol version: 34
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV1Serializer.readVersion(RecordDataV1Serializer.java:2010)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV2Serializer.readMvccDataEntry(RecordDataV2Serializer.java:328)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV2Serializer.readPlainRecord(RecordDataV2Serializer.java:162)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV1Serializer.readRecord(RecordDataV1Serializer.java:208)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV2Serializer$2.readWithHeaders(RecordV2Serializer.java:163)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV1Serializer.readWithCrc(RecordV1Serializer.java:374)
> ... 19 more
> Suppressed: class org.apache.ignite.internal.processors.cache.persistence.wal.crc.IgniteDataIntegrityViolationException: val: 1816937838 writtenCrc: 1387698722
> at org.apache.ignite.internal.processors.cache.persistence.wal.io.FileInput$Crc32CheckingFileInput.close(FileInput.java:106)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV1Serializer.readWithCrc(RecordV1Serializer.java:381)
> ... 19 more
> Caused by: class org.apache.ignite.IgniteCheckedException: Unsupported protocol version: 34
> at org.apache.ignite.internal.processors.cache.persistence.tree.io.CacheVersionIO.checkProtocolVersion(CacheVersionIO.java:143)
> at org.apache.ignite.internal.processors.cache.persistence.tree.io.CacheVersionIO.readSize(CacheVersionIO.java:156)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV1Serializer.readVersion(RecordDataV1Serializer.java:2003)
> ... 24 more
> [23:16:06,299][SEVERE][main][IgniteKernal] Got exception while starting (will rollback startup routine).
> class org.apache.ignite.IgniteCheckedException: Failed to read WAL record at position: 27700852 size: 67108864
> at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.handleRecordException(AbstractWalRecordsIterator.java:305)
> at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.handleRecordException(FileWriteAheadLogManager.java:2691)
> at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.advanceRecord(AbstractWalRecordsIterator.java:269)
> at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.advance(AbstractWalRecordsIterator.java:158)
> at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.onNext(AbstractWalRecordsIterator.java:127)
> at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.onNext(AbstractWalRecordsIterator.java:53)
> at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.nextX(GridCloseableIteratorAdapter.java:41)
> at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.tailPointer(GridCacheDatabaseSharedManager.java:1984)
> at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.startMemoryRestore(GridCacheDatabaseSharedManager.java:1958)
> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1050)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2040)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1732)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
> at org.apache.ignite.Ignition.start(Ignition.java:348)
> at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read WAL record at position: 27700852 size: 67108864
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV1Serializer.readWithCrc(RecordV1Serializer.java:395)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV2Serializer.readRecord(RecordV2Serializer.java:236)
> at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.advanceRecord(AbstractWalRecordsIterator.java:254)
> ... 17 more
> Caused by: java.io.IOException: class org.apache.ignite.IgniteCheckedException: Unsupported protocol version: 34
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV1Serializer.readVersion(RecordDataV1Serializer.java:2010)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV2Serializer.readMvccDataEntry(RecordDataV2Serializer.java:328)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV2Serializer.readPlainRecord(RecordDataV2Serializer.java:162)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV1Serializer.readRecord(RecordDataV1Serializer.java:208)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV2Serializer$2.readWithHeaders(RecordV2Serializer.java:163)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV1Serializer.readWithCrc(RecordV1Serializer.java:374)
> ... 19 more
> Suppressed: class org.apache.ignite.internal.processors.cache.persistence.wal.crc.IgniteDataIntegrityViolationException: val: 1816937838 writtenCrc: 1387698722
> at org.apache.ignite.internal.processors.cache.persistence.wal.io.FileInput$Crc32CheckingFileInput.close(FileInput.java:106)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV1Serializer.readWithCrc(RecordV1Serializer.java:381)
> ... 19 more
> Caused by: class org.apache.ignite.IgniteCheckedException: Unsupported protocol version: 34
> at org.apache.ignite.internal.processors.cache.persistence.tree.io.CacheVersionIO.checkProtocolVersion(CacheVersionIO.java:143)
> at org.apache.ignite.internal.processors.cache.persistence.tree.io.CacheVersionIO.readSize(CacheVersionIO.java:156)
> at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV1Serializer.readVersion(RecordDataV1Serializer.java:2003)
> ... 24 more
> [23:16:06,302][WARNING][main][IgniteKernal] Attempt to stop starting grid. This operation cannot be guaranteed to be successful.
> [23:16:06,313][INFO][main][GridTcpRestProtocol] Command protocol successfully stopped: TCP binary
> [23:16:06,347][SEVERE][ttl-cleanup-worker-#45][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Got interrupted while waiting for future to complete.]]
> class org.apache.ignite.IgniteException: Got interrupted while waiting for future to complete.
> at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.localJoin(GridDiscoveryManager.java:2208)
> at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:136)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.internal.IgniteInterruptedCheckedException: Got interrupted while waiting for future to complete.
> at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:186)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
> at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.localJoin(GridDiscoveryManager.java:2205)
> ... 3 more
> [23:16:06,347][WARNING][ttl-cleanup-worker-#45][FailureProcessor] No deadlocked threads detected.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)