[jira] [Created] (IGNITE-29185) cpMapSnapshot.bin written by Ignite 2.17/2.18 can't be read by current version (generated IDTO serializer changed UUID format)

["Nikolay Izhikov (Jira)" <ji...@apache.org>]

Nikolay Izhikov created IGNITE-29185:
----------------------------------------

             Summary: cpMapSnapshot.bin written by Ignite 2.17/2.18 can't be read by current version (generated IDTO serializer changed UUID format)
                 Key: IGNITE-29185
                 URL: https://issues.apache.org/jira/browse/IGNITE-29185
             Project: Ignite
          Issue Type: Bug
    Affects Versions: 2.19
            Reporter: Nikolay Izhikov
         Attachments: IGNITE-29185-reproducer.patch

h3. Problem

{{cpMapSnapshot.bin}} (earliest checkpoint map snapshot, see {{CheckpointMarkersStorage#EARLIEST_CP_SNAPSHOT_FILE}}) written by Ignite 2.17 / 2.18 can't be deserialized by current master (future 2.19).

The file is an {{EarliestCheckpointMapSnapshot}} marshalled with {{JdkMarshaller}}. In 2.17 and 2.18 the payload is written by hand:

{code:java}
U.writeMap(out, data);              // keys (UUID) and values via out.writeObject(...)
U.writeCollection(out, checkpointIds); // UUID elements via out.writeObject(...)
{code}

Since IGNITE-27709 the class is serialized by generated code, and since IGNITE-27775 the generator emits typed code for collections. The generated {{EarliestCheckpointMapSnapshotSerializer}} now writes/reads the outer map keys and the {{checkpointIds}} elements with {{U.writeUuid / U.readUuid}} (null flag + two longs) instead of {{writeObject / readObject}} (Java-serialized {{java.util.UUID}}). The two formats are not interchangeable, so the file produced by a previous version can't be read after upgrade.

The header change made by IGNITE-24260 (protocol version byte removed) is *not* a problem: the reader masks the version byte.

h3. Impact

{{CheckpointMarkersStorage#initialize}} catches the failure, logs

{noformat}
Failed to unmarshal earliest checkpoint map snapshot
{noformat}

deletes the file and rebuilds the earliest checkpoint map from checkpoint markers. No data is lost, but the optimization is silently lost on the first start after an upgrade and the node logs an error. PDS files must stay backward compatible, so the old format has to be readable.

h3. Reproducer

{{CheckpointMapSnapshotCompatibilityTest}} in {{ignite-compatibility}} (attached): starts a 2.17.0 / 2.18.0 node with persistence, waits for {{cpMapSnapshot.bin}}, restarts on the same PDS with the current version and asserts the error above isn't logged.

Test output (same for 2.17.0 and 2.18.0):

{noformat}
class org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: null
	at org.apache.ignite.marshaller.jdk.JdkMarshallerImpl.unmarshal0(JdkMarshallerImpl.java:128)
	at org.apache.ignite.marshaller.jdk.JdkMarshallerImpl.unmarshal0(JdkMarshallerImpl.java:135)
	at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:74)
	at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointMarkersStorage.initialize(CheckpointMarkersStorage.java:209)
	at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointManager.initializeStorage(CheckpointManager.java:325)
	at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:854)
	at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:3075)
	at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1122)
	at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1713)
	at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1635)
	at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1077)
	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:587)
	at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:1307)
	at org.apache.ignite.compatibility.testframework.junits.IgniteCompatibilityAbstractTest.startGrid(IgniteCompatibilityAbstractTest.java:344)
	at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:1246)
	at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:1222)
	at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:1053)
	at org.apache.ignite.compatibility.persistence.CheckpointMapSnapshotCompatibilityTest.testSnapshotFileReadableByCurrentVersion(CheckpointMapSnapshotCompatibilityTest.java:121)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.apache.ignite.testframework.junits.GridAbstractTest$6.run(GridAbstractTest.java:2464)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.EOFException
	at java.base/java.io.ObjectInputStream$BlockDataInputStream.readBoolean(ObjectInputStream.java:3374)
	at java.base/java.io.ObjectInputStream.readBoolean(ObjectInputStream.java:1062)
	at org.apache.ignite.internal.dto.IgniteDataTransferObjectInput.readBoolean(IgniteDataTransferObjectInput.java:99)
	at org.apache.ignite.internal.util.CommonUtils.readUuid(CommonUtils.java:682)
	at org.apache.ignite.internal.processors.cache.persistence.checkpoint.EarliestCheckpointMapSnapshotSerializer.readExternal(EarliestCheckpointMapSnapshotSerializer.java:84)
	at org.apache.ignite.internal.processors.cache.persistence.checkpoint.EarliestCheckpointMapSnapshotSerializer.readExternal(EarliestCheckpointMapSnapshotSerializer.java:38)
	at org.apache.ignite.internal.dto.IgniteDataTransferObject.readIgniteDataTransferObject(IgniteDataTransferObject.java:84)
	at org.apache.ignite.internal.dto.IgniteDataTransferObject.readExternal(IgniteDataTransferObject.java:67)
	at java.base/java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2306)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2255)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1733)
	at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:509)
	at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:467)
	at org.apache.ignite.marshaller.jdk.JdkMarshallerImpl.unmarshal0(JdkMarshallerImpl.java:120)
	... 27 more
{noformat}

h3. Possible fix

Make the generated code (or a hand-written {{readExternalData}} in {{EarliestCheckpointMapSnapshot}}) able to read the legacy layout, e.g. keep {{writeObject/readObject}} for this class or detect the old format. Add the compatibility test to the suite.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)