You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Prasad Bhalerao <pr...@gmail.com> on 2019/01/11 09:35:56 UTC
Re: After upgrading 2.7 getting error during unmarshalling (Works
fine on 2.6)
Hi Ilya,
I am not able reproduce this issue in small reproducer project but this is
consistently happening in my project. So I debugged this issue and attached
the screenshot in this mail.
*NOTE: *This issue occurs if the statistics are enabled on cache
configuration level [cacheCfg.setStatisticsEnabled(true)].
As shown in screenshots there 22 cache metrics in cacheMetrics hashmap. All
these cache metrics gets serialized successfully on server node. But at the
time of deserialization on client node only first metrics get de-serialized
successfully but all other metrics till iteration count 13 are
de-serialized as null value and on iteration 14 “ref” byte value in
“OptimizedObjectInputStream.readObject0()” method is read as 81 and code
throws exception.
I think this where it is going wrong. The object copy at the time of
serialization and de-serialization should be same but that’s not happening
in Ignite 2.7 version.
So I debugged this on Ignite 2.6 version. On 2.6 version all 22
cacheMetrics are being de-serialized successfully.
This Looks like a bug to me in serializer/deseriliazer code.
*AffinityJob result being serialized on server:*
*AffinityJob result being de-serialized on client:*
Thanks,
Prasad
On Wed, Jan 9, 2019 at 6:41 PM Ilya Kasnacheev <il...@gmail.com>
wrote:
> Hello!
>
> Do you have a reproducer project to reliably confirm this issue?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 9 янв. 2019 г. в 12:39, Akash Shinde <ak...@gmail.com>:
>
>> Added dev@ignite.apache.org.
>>
>> Should I log Jira for this issue?
>>
>> Thanks,
>> Akash
>>
>>
>>
>> On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde <ak...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > No both nodes, client and server are running on Ignite 2.7 version. I am
>> > starting both server and client from Intellij IDE.
>> >
>> > Version printed in Server node log:
>> > Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>> >
>> > Version in client node log:
>> > Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>> >
>> > Thanks,
>> > Akash
>> >
>> > On Tue, Jan 8, 2019 at 5:18 PM Mikael <mi...@telia.com>
>> wrote:
>> >
>> >> Hi!
>> >>
>> >> Any chance you might have one node running 2.6 or something like that ?
>> >>
>> >> It looks like it get a different object that does not match the one
>> >> expected in 2.7
>> >>
>> >> Mikael
>> >> Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
>> >>
>> >> Before submitting the affinity task ignite first gets the affinity
>> cached
>> >> function (AffinityInfo) by submitting the cluster wide task
>> "AffinityJob".
>> >> But while in the process of retrieving the output of this AffinityJob,
>> >> ignite deserializes this output. I am getting exception while
>> deserailizing
>> >> this output.
>> >> In TcpDiscoveryNode.readExternal() method while deserailizing the
>> >> CacheMetrics object from input stream on 14th iteration I am getting
>> >> following exception. Complete stack trace is given in this mail chain.
>> >>
>> >> Caused by: java.io.IOException: Unexpected error occurred during
>> >> unmarshalling of an instance of the class:
>> >> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>> >>
>> >> This is working fine on Ignite 2.6 version but giving problem on 2.7.
>> >>
>> >> Is this a bug or am I doing something wrong?
>> >>
>> >> Can someone please help?
>> >>
>> >> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde <ak...@gmail.com>
>> >> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> When execute affinity.partition(key), I am getting following exception
>> >>> on Ignite 2.7.
>> >>>
>> >>> Stacktrace:
>> >>>
>> >>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR
>> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>> >>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>> >>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>> >>> isCancelled=false, retry=null]
>> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>> >>> with optimized marshaller
>> >>> at
>> >>>
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>> >>> at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>> >>> at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>> >>> at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>> >>> at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>> >>> at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>> >>> at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>> >>> at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>> >>> at
>> >>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> >>> at
>> >>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> >>> at java.lang.Thread.run(Thread.java:748)
>> >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>> >>> unmarshal object with optimized marshaller
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>> >>> at
>> >>>
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>> >>> at
>> >>>
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>> >>> ... 10 common frames omitted
>> >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>> >>> deserialize object with given class loader:
>> >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>> >>> deserialize object
>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>> >>> at
>> >>>
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>> >>> ... 16 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>> >>> ... 18 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>>
>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>> ... 21 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize field
>> >>> [name=assignment]
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>> ... 27 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>>
>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>> >>> ... 30 common frames omitted
>> >>> Caused by: java.io.IOException: Unexpected error occurred during
>> >>> unmarshalling of an instance of the class:
>> >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>> Check
>> >>> that all nodes are running the same version of Ignite and that all
>> nodes
>> >>> have GridOptimizedMarshaller configured with identical optimized
>> classes
>> >>> lists, if any (see setClassNames and setClassNamesPath methods). If
>> your
>> >>> serialized classes implement java.io.Externalizable interface, verify
>> that
>> >>> serialization logic is correct.
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>> ... 41 common frames omitted
>> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>> >>> qzkpr01.p13.eng.in03.com:50024)] INFO
>> org.apache.zookeeper.ClientCnxn
>> >>> - Opening socket connection to server
>> >>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024. Will not attempt to
>> >>> authenticate using SASL (unknown error)
>> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>> >>> qzkpr01.p13.eng.in03.com:50024)] INFO
>> org.apache.zookeeper.ClientCnxn
>> >>> - Socket connection established to
>> >>> qzkpr01.p13.eng.in03.com/10.114.113.201:50024, initiating session
>> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>> >>> qzkpr01.p13.eng.in03.com:50024)] WARN
>> org.apache.zookeeper.ClientCnxn
>> >>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7
>> has
>> >>> expired
>> >>> 2019-01-07 21:23:03,140 6699925 [main-SendThread(
>> >>> qzkpr01.p13.eng.in03.com:50024)] INFO
>> org.apache.zookeeper.ClientCnxn
>> >>> - Unable to reconnect to ZooKeeper service, session 0x3673a59f5dd01c7
>> has
>> >>> expired, closing socket connection
>> >>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>> >>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (Expired)
>> >>> 2019-01-07 21:23:03,140 6699925 [main-EventThread] INFO
>> >>> org.apache.zookeeper.ZooKeeper - Initiating client connection,
>> >>> connectString=qzkpr01.p13.eng.in03.com:50024,
>> >>> qzkpr02.p13.eng.in03.com:50024,qzkpr03.p13.eng.in03.com:50024/qkafka
>> >>> sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@3003d288
>> >>> 2019-01-07 21:23:03,156 6699941 [main-EventThread] INFO
>> >>> org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
>> >>> 0x3673a59f5dd01c7
>> >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>> >>> qzkpr03.p13.eng.in03.com:50024)] INFO
>> org.apache.zookeeper.ClientCnxn
>> >>> - Opening socket connection to server
>> >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024. Will not attempt to
>> >>> authenticate using SASL (unknown error)
>> >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>> >>> qzkpr03.p13.eng.in03.com:50024)] INFO
>> org.apache.zookeeper.ClientCnxn
>> >>> - Socket connection established to
>> >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, initiating session
>> >>> 2019-01-07 21:23:03,296 6700081 [main-SendThread(
>> >>> qzkpr03.p13.eng.in03.com:50024)] INFO
>> org.apache.zookeeper.ClientCnxn
>> >>> - Session establishment complete on server
>> >>> qzkpr03.p13.eng.in03.com/10.114.113.203:50024, sessionid =
>> >>> 0x3673a59f5dd01ce, negotiated timeout = 10000
>> >>> 2019-01-07 21:23:03,296 6700081 [main-EventThread] INFO
>> >>> org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
>> >>> 2019-01-07 21:23:03,609 6700394 [mgmt-#68%springDataNode%] ERROR
>> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>> >>> sesId=d778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>> >>> jobId=e778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>> >>> isCancelled=false, retry=null]
>> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>> >>> with optimized marshaller
>> >>> at
>> >>>
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>> >>> at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>> >>> at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>> >>> at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>> >>> at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>> >>> at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>> >>> at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>> >>> at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>> >>> at
>> >>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> >>> at
>> >>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> >>> at java.lang.Thread.run(Thread.java:748)
>> >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>> >>> unmarshal object with optimized marshaller
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>> >>> at
>> >>>
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>> >>> at
>> >>>
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>> >>> ... 10 common frames omitted
>> >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>> >>> deserialize object with given class loader:
>> >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>> >>> deserialize object
>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>> >>> at
>> >>>
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>> >>> ... 16 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>> >>> ... 18 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>>
>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>> ... 21 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize field
>> >>> [name=assignment]
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>> ... 27 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>>
>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>> >>> ... 30 common frames omitted
>> >>> Caused by: java.io.IOException: Unexpected error occurred during
>> >>> unmarshalling of an instance of the class:
>> >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>> Check
>> >>> that all nodes are running the same version of Ignite and that all
>> nodes
>> >>> have GridOptimizedMarshaller configured with identical optimized
>> classes
>> >>> lists, if any (see setClassNames and setClassNamesPath methods). If
>> your
>> >>> serialized classes implement java.io.Externalizable interface, verify
>> that
>> >>> serialization logic is correct.
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>> ... 41 common frames omitted
>> >>> 2019-01-07 21:23:04,140 6700925 [mgmt-#69%springDataNode%] ERROR
>> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>> >>> sesId=f778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>> >>> jobId=0878e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>> >>> isCancelled=false, retry=null]
>> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>> >>> with optimized marshaller
>> >>> at
>> >>>
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>> >>> at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>> >>> at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>> >>> at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>> >>> at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>> >>> at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>> >>> at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>> >>> at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>> >>> at
>> >>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> >>> at
>> >>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> >>> at java.lang.Thread.run(Thread.java:748)
>> >>> Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to
>> >>> unmarshal object with optimized marshaller
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>> >>> at
>> >>>
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>> >>> at
>> >>>
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>> >>> ... 10 common frames omitted
>> >>> Caused by: org.apache.ignite.IgniteCheckedException: Failed to
>> >>> deserialize object with given class loader:
>> >>> [clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to
>> >>> deserialize object
>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
>> >>> at
>> >>>
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>> >>> at
>> >>>
>> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
>> >>> ... 16 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>> [typeName=org.apache.ignite.internal.util.lang.GridTuple3]
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
>> >>> ... 18 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>>
>> [typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>> ... 21 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize field
>> >>> [name=assignment]
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:954)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>> ... 27 common frames omitted
>> >>> Caused by: java.io.IOException: Failed to deserialize object
>> >>>
>> [typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
>> >>> ... 30 common frames omitted
>> >>> Caused by: java.io.IOException: Unexpected error occurred during
>> >>> unmarshalling of an instance of the class:
>> >>> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>> Check
>> >>> that all nodes are running the same version of Ignite and that all
>> nodes
>> >>> have GridOptimizedMarshaller configured with identical optimized
>> classes
>> >>> lists, if any (see setClassNames and setClassNamesPath methods). If
>> your
>> >>> serialized classes implement java.io.Externalizable interface, verify
>> that
>> >>> serialization logic is correct.
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
>> >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:425)
>> >>> at
>> >>>
>> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:949)
>> >>> at
>> >>>
>> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
>> >>> ... 41 common frames omitted
>> >>>
>> >>> Thanks,
>> >>> Akash
>> >>>
>> >>
>>
>