You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "Calvin KL Wong, CLSA" <ca...@clsa.com> on 2018/06/27 01:13:41 UTC
Best practice for class versioning: marshaller error
Hi,
I am running Ignite 2.3 using Cassandra as my persistence store.
I got unmarshall error when a server node trying to unmarshall an object of old version from Cassandra.
This is the scenario:
1. Object of ClassA (older version) is serialized and persisted into Cassandra
2. ClassA is updated with new fields
3. Server node's classpath has new ClassA class file.
4. Server node is deployed to the Ignite grid. It tries to deserialize from the serialized object of ClassA (older version) and results the result to a query. Then got this exception:
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager$CacheClassLoader@3bc180c9, err=Unexpected error occurred during unmarshalling of an instance of the class: java.time.Ser. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.]
at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:236) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at
I believe this use case is pretty normal.
Do you have any recommendation/best practice on how I can avoid this error? Is BinaryObject able to prevent this issue from happening?
Thanks,
Calvin
Calvin KL Wong
Sr. Lead Engineer, Execution Services
D +852 2600 7983 | M +852 9267 9471 | T +852 2600 8888
5/F, One Island East, 18 Westlands Road, Island East, Hong Kong
[:1. Social Media Icons:CLSA_Social Media Icons_linkedin.png]<https://hk.linkedin.com/company/clsa>[:1. Social Media Icons:CLSA_Social Media Icons_twitter.png]<https://twitter.com/clsainsights?lang=en>[:1. Social Media Icons:CLSA_Social Media Icons_youtube.png]<https://www.youtube.com/channel/UC0qWp_lLnOcRYmBlCNQgZKA>[:1. Social Media Icons:CLSA_Social Media Icons_facebook.png]<https://www.facebook.com/clsacommunity/>
clsa.com<https://www.clsa.com/>
Insights. Liquidity. Capital.
[CLSA_RGB]<https://www.clsa.com/member>
A CITIC Securities Company
The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.
RE: Best practice for class versioning: marshaller error
Posted by "Calvin KL Wong, CLSA" <ca...@clsa.com>.
Got it, thanks for the info.
Thanks,
Calvin
-----Original Message-----
From: dkarachentsev [mailto:dkarachentsev@gridgain.com]
Sent: Monday, July 16, 2018 9:57 PM
To: user@ignite.apache.org
Subject: RE: Best practice for class versioning: marshaller error
Hi Calvin,
> Can I assume that BinaryMarshaller won't be used for any object embedded
> inside GridCacheQueryResponse?
Yes, because Binary can fallback to Optimized, but not vice versa.
> If I am correct, do you have any suggestion on how I can avoid this type
> of issue?
Probably you need to avoid using incorrectly serialized objects, or
implement your own Externalizable, when you will do manual
serializing/deserializing of such fields.
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.
RE: Best practice for class versioning: marshaller error
Posted by dkarachentsev <dk...@gridgain.com>.
Hi Calvin,
> Can I assume that BinaryMarshaller won't be used for any object embedded
> inside GridCacheQueryResponse?
Yes, because Binary can fallback to Optimized, but not vice versa.
> If I am correct, do you have any suggestion on how I can avoid this type
> of issue?
Probably you need to avoid using incorrectly serialized objects, or
implement your own Externalizable, when you will do manual
serializing/deserializing of such fields.
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Best practice for class versioning: marshaller error
Posted by "Calvin KL Wong, CLSA" <ca...@clsa.com>.
Then I am confused with the exception I got. Please bear with me.
I believe what happened was a client (with a new version of user class) trying to deserialize an old object (an old version of user class) from Continuous Query.
This was what happened base on the log:
1. GridCacheIoManager unmarshalled GridCacheQueryResponse
2. GridCacheQueryResponse unmarshalled one element in its bytes collection
3. BinaryMarshaller (internally GridBinaryMarshaller) was being used
1. It then found out that it needed to unmarshal an Optimized object
1. The Optimized object turned out to be a GridCacheQueryResponseEntry and tried to unmarshal GridCacheQueryResponseEntry.val
1. Looks like we tried to unmarshall serveral 'Serializable' objects along the way
OptimizedMarshaller is always used for all system type (unless it is Binarylizable). I looked at the OptimizedObjectInputStream.java, I cannot find any way to 'read object using BinaryMarshaller or BinaryReaderExImpl ' once we are inside OptimizedObjectInputStream.readObject0().
Can I assume that BinaryMarshaller won't be used for any object embedded inside GridCacheQueryResponse?
If I am correct, do you have any suggestion on how I can avoid this type of issue? Hopefully I am wrong.
Thanks,
Calvin
Caused by: java.io.IOException: Unexpected error occurred during unmarshalling of an instance of the class: java.time.Ser. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:359) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:365) ~[?:1.8.0_74]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:513) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:601) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] <<<<<<<<<<<<<<<<< Step 3.1.1.1
at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:927) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:365) ~[?:1.8.0_74]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:513) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:601) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] <<<<<<<<<<<<<<<<< Step 3.1.1.1
at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:927) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:365) ~[?:1.8.0_74]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:513) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:601) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] <<<<<<<<<<<<<<<<< Step 3.1.1.1
at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:927) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:365) ~[?:1.8.0_74]
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponseEntry.readExternal(GridCacheQueryResponseEntry.java:90) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] <<<<<<<<<<<<<<<<< Step 3.1.1
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:545) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:917) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:365) ~[?:1.8.0_74]
at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:227) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1783) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] <<<<<<<<<<<<<<<<< Step 3.1
at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1962) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:310) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] <<<<<<<<<<<<<<<<< Step 3
at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:99) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponse.unmarshalCollection0(GridCacheQueryResponse.java:189) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] <<<<<<<<<<<<<<<<< Step 2
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponse.finishUnmarshal(GridCacheQueryResponse.java:162) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1526) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] <<<<<<<<<<<<<<<<< Step 1
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:574) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
-----Original Message-----
From: slava.koptilin [mailto:slava.koptilin@gmail.com]
Sent: Friday, June 29, 2018 1:04 AM
To: user@ignite.apache.org
Subject: RE: Best practice for class versioning: marshaller error
Hello,
In that case, only 'createdTime' fall back to OptimizedMarshaller.
Thanks,
Slava.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.
RE: Best practice for class versioning: marshaller error
Posted by "slava.koptilin" <sl...@gmail.com>.
Hello,
In that case, only 'createdTime' fall back to OptimizedMarshaller.
Thanks,
Slava.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Best practice for class versioning: marshaller error
Posted by "Calvin KL Wong, CLSA" <ca...@clsa.com>.
Thanks, Dmitry.
>>2-3. In your case, you have and java.time.Ser in one of the fields of your POJO (or maybe inside of depended object), and it is Externalizable. In such case BinaryMarshalelr falls back to OptimizedMarshaller with all the issues. Try to remove it from your POJOs or make transient.
Given that I have this class:
class NewOrderRequest {
int securityId;
Instant createdTime;
}
Instant has a writeReplace method that returns java.time.Ser.
Is the whole 'NewOrderRequest' fall back to OptimizedMarshaller? Or only 'createdTime' fall back to OptimizedMarshaller.
-Calvin
-----Original Message-----
From: dkarachentsev [mailto:dkarachentsev@gridgain.com]
Sent: Thursday, June 28, 2018 2:59 PM
To: user@ignite.apache.org
Subject: RE: Best practice for class versioning: marshaller error
Hi Calvin,
1. Enlist I mean that if you want, for example, to get to see what fields
present in BinaryObject. In other words, if you want to work with
BinaryObject directly. For POJO serialization/deserialization this should
not be and issue at all.
2-3. In your case, you have and java.time.Ser in one of the fields of your
POJO (or maybe inside of depended object), and it is Externalizable. In such
case BinaryMarshalelr falls back to OptimizedMarshaller with all the issues.
Try to remove it from your POJOs or make transient.
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.
RE: Best practice for class versioning: marshaller error
Posted by dkarachentsev <dk...@gridgain.com>.
Hi Calvin,
1. Enlist I mean that if you want, for example, to get to see what fields
present in BinaryObject. In other words, if you want to work with
BinaryObject directly. For POJO serialization/deserialization this should
not be and issue at all.
2-3. In your case, you have and java.time.Ser in one of the fields of your
POJO (or maybe inside of depended object), and it is Externalizable. In such
case BinaryMarshalelr falls back to OptimizedMarshaller with all the issues.
Try to remove it from your POJOs or make transient.
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Best practice for class versioning: marshaller error
Posted by "Calvin KL Wong, CLSA" <ca...@clsa.com>.
Hi Dmitry,
Thanks for your response!
I want to confirm my understanding -
If I disable compact footer and add your POJOs in BinaryConfiguration, I will be deserialize back the serialized object after I stop grid and removed the local files that hold the marshaller cache (I believe they are located in the tmp folder).
Questions:
1. >> Second case, if you by some reason want to use BinaryObject directly. You should be aware that fact you can read and add fields, but you will not be able to enlist them..... Again, hopefully, you can solve this with BinaryConfiguration
I don't get the 'enlist' part in this sentence. Does that mean I cannot turn them back into POJO even if I list my POJO in BinaryConfiguration? Or this has something to do with BinaryObject.
2. Regarding BinaryObject, I have a regular POJO user class that does not implement Binarylizable and Externalizable, and has no 'writeObject' and 'readObject' method (BinaryContext.canUseReflectiveSerializer). I see that its BinaryClassDescriptor has a mode of BinaryWriteMode.OBJECT, not BinaryWriteMode.BINARY_OBJ. Does not mean I cannot use it as BinaryObject? From reading the doc, looks like all sample objects will be serialized as BinaryObject automatically (https://apacheignite.readme.io/docs/binary-marshaller). However, that is not what I am seeing in the code.
3. The error log shows that OptimizedMarshaller is being called. Since all my classes are neither Binarylizable nor Externalizable, does that mean the marshalling error is related to versioning of an Ignite class?
Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager$CacheClassLoader@3bc180c9, err=Unexpected error occurred during unmarshalling of an instance of the class: java.time.Ser. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.]
at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:236) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59]
at
Please note that I ask these questions because I need to look into an issue that happened some time ago. I have the log with me, but I am not able to reproduce the issue. That's why I try to understand more.
Thanks,
Calvin
-----Original Message-----
From: dkarachentsev [mailto:dkarachentsev@gridgain.com]
Sent: Wednesday, June 27, 2018 8:52 PM
To: user@ignite.apache.org
Subject: Re: Best practice for class versioning: marshaller error
Hi Calvin,
BinaryMarshaller can solve that issue with involving a few more.
First of all, you will need to disable compact footer to let each
BinaryObject has it's schema in footer.
If you need just put/get POJOs everything will be fine. But you need to
enlist your POJO in BinaryConfiguration [1], because Ignite identifies type
with typeId, that is by default hash from full class name. So each
BinaryObject keeps only typeId. To find proper class to which it should be
deserialized, Ignite needs mapping class name -> typeId. This mapping keeps
in marshaller cache that holds in memory and in local files. Anyway you can
stop grid, remove Ignite files and ran into issue that you cannot
deserialize object, because Ignite cannot find class name for some typeId.
And as I said, you need to enlist this class in BinaryConfiguration, that
Ignite will read on start and fill marshaller cache.
Second case, if you by some reason want to use BinaryObject directly. You
should be aware that fact you can read and add fields, but you will not be
able to enlist them. Because here used the same approach: field name
converts to fieldId. Mapping fieldName -> fieldId is called meta info and
stored in local meta cache. Again, hopefully, you can solve this with
BinaryConfiguration.
In summary, BinaryMarshaller is OK with different schema, but you need
disable compact footer and add your POJOs in BinaryConfiguration.
[1]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/BinaryConfiguration.html
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon request.
Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.
Re: Best practice for class versioning: marshaller error
Posted by dkarachentsev <dk...@gridgain.com>.
Hi Calvin,
BinaryMarshaller can solve that issue with involving a few more.
First of all, you will need to disable compact footer to let each
BinaryObject has it's schema in footer.
If you need just put/get POJOs everything will be fine. But you need to
enlist your POJO in BinaryConfiguration [1], because Ignite identifies type
with typeId, that is by default hash from full class name. So each
BinaryObject keeps only typeId. To find proper class to which it should be
deserialized, Ignite needs mapping class name -> typeId. This mapping keeps
in marshaller cache that holds in memory and in local files. Anyway you can
stop grid, remove Ignite files and ran into issue that you cannot
deserialize object, because Ignite cannot find class name for some typeId.
And as I said, you need to enlist this class in BinaryConfiguration, that
Ignite will read on start and fill marshaller cache.
Second case, if you by some reason want to use BinaryObject directly. You
should be aware that fact you can read and add fields, but you will not be
able to enlist them. Because here used the same approach: field name
converts to fieldId. Mapping fieldName -> fieldId is called meta info and
stored in local meta cache. Again, hopefully, you can solve this with
BinaryConfiguration.
In summary, BinaryMarshaller is OK with different schema, but you need
disable compact footer and add your POJOs in BinaryConfiguration.
[1]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/BinaryConfiguration.html
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/