Handling collections in BinaryMarshaller

[Vladimir Ozerov <vo...@gridgain.com>]

Folks,

We have some very strange logic for collections in BinaryMarshaller:
https://issues.apache.org/jira/browse/IGNITE-2099
Instead of writing them as any other objects, we treat them specially and
write in unified form:
[collection_flag] + [collection_type_flag] + [size] + [items].
I remember we introduced this in good old GridGain days to handle
interoperability issues between Java and .NET.

This design is badly wrong:
1) Users cannot use their custom collections without following some
ridiculous requirements.
2) We loose some important collection-specific data. E.g., comparators in
sorted collections, load factor, etc..

I think we must remove all this weird logic and always write collections as
any other objects. The only exception are
BinaryReader.readCollection/readMap and
BinaryWriter.writeCollection/writeMap methods. In this case we write
collections in unified format and it is user responsibility to explain how
exactly to deserialize such fields using either some factory or explicitly
provided class name.

If user needs interoperability - sorry, Java TreeMap is not interoperable
with .NET SortedDictionary - these are different types. But user can always
create a *wrapper over collection* and make it interoperable.

Thoughts?

Vladimir.

Re: Handling collections in BinaryMarshaller

[Vladimir Ozerov <vo...@gridgain.com>]

I am going to drop support of the following classes. All these classes are
currently written in a special unified way. After my changes they will be
written *as any other usual object*.

1) Sorted collections:

   - TreeSet
   - TreeMap
   - ConcurrentSkipListSet

Reason: currently we do not write Comparator which might lead to exceptions
on the other side.

2) Special collections:

   - ConcurrentLinkedQueue
   - ConcurrentHashMap
   - PropertiesMap

Reason: too specific use cases.

3) Map.Entry
Reason: this is an interface with billion implemenations, we cannot
unmarshal it correctly on the other side without full serialization of
concrete class.

On Wed, Dec 9, 2015 at 10:57 AM, Vladimir Ozerov <vo...@gridgain.com>
wrote:

> Dmitriy,
> Correct. This is what I meant.
>
> On Tue, Dec 8, 2015 at 10:26 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
>> Vladimir,
>>
>> I believe the default collections in Java and .NET should be supported out
>> of the box. Moreover, if we know the collection type, e.g. HashMap, we can
>> always provide a more efficient way of serializing it ourselves, in the
>> Binary marshaller.
>>
>> Is this something you had in mind, or were you proposing something
>> different.
>>
>> D.
>>
>> On Tue, Dec 8, 2015 at 6:29 AM, Vladimir Ozerov <vo...@gridgain.com>
>> wrote:
>>
>> > Alex,
>> >
>> > What interface do you mean? If user has collection field in class and
>> > explicitly call BinaryWriter.writeCollection(), we can leave current
>> > interoperability support - it is not a problem.
>> > As per your second point - user could pass collections e.g. as argument
>> to
>> > Java task started from .NET. This is where we will loose
>> interoperabiltiy
>> > and will force user to create some wrappers. But these are very specific
>> > use cases.
>> >
>> > BTW, proposed solution is almost exactly how we work with collections in
>> > .NET.
>> >
>> > On Tue, Dec 8, 2015 at 4:57 PM, Alexey Goncharuk <
>> > alexey.goncharuk@gmail.com
>> > > wrote:
>> >
>> > > I like the idea, however it has obvious downsides. First, if a user
>> class
>> > > contains a collection, we force user to implement additional
>> interface,
>> > > even if the collection is a simple ArrayList. Second, I do not see how
>> > this
>> > > plain collection can be the value for the cache - user will always
>> need
>> > to
>> > > write a wrapper/containing class around it.
>> > >
>> > > I think we should provide minimum support for basic types - HashMap,
>> > > LinkedHashMap, ArrayList and treat other classes the way Vladimir
>> > > described.
>> > >
>> >
>>
>
>

Re: Handling collections in BinaryMarshaller

[Vladimir Ozerov <vo...@gridgain.com>]

Dmitriy,
Correct. This is what I meant.

On Tue, Dec 8, 2015 at 10:26 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:

> Vladimir,
>
> I believe the default collections in Java and .NET should be supported out
> of the box. Moreover, if we know the collection type, e.g. HashMap, we can
> always provide a more efficient way of serializing it ourselves, in the
> Binary marshaller.
>
> Is this something you had in mind, or were you proposing something
> different.
>
> D.
>
> On Tue, Dec 8, 2015 at 6:29 AM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
>
> > Alex,
> >
> > What interface do you mean? If user has collection field in class and
> > explicitly call BinaryWriter.writeCollection(), we can leave current
> > interoperability support - it is not a problem.
> > As per your second point - user could pass collections e.g. as argument
> to
> > Java task started from .NET. This is where we will loose interoperabiltiy
> > and will force user to create some wrappers. But these are very specific
> > use cases.
> >
> > BTW, proposed solution is almost exactly how we work with collections in
> > .NET.
> >
> > On Tue, Dec 8, 2015 at 4:57 PM, Alexey Goncharuk <
> > alexey.goncharuk@gmail.com
> > > wrote:
> >
> > > I like the idea, however it has obvious downsides. First, if a user
> class
> > > contains a collection, we force user to implement additional interface,
> > > even if the collection is a simple ArrayList. Second, I do not see how
> > this
> > > plain collection can be the value for the cache - user will always need
> > to
> > > write a wrapper/containing class around it.
> > >
> > > I think we should provide minimum support for basic types - HashMap,
> > > LinkedHashMap, ArrayList and treat other classes the way Vladimir
> > > described.
> > >
> >
>

Re: Handling collections in BinaryMarshaller

[Dmitriy Setrakyan <ds...@apache.org>]

Vladimir,

I believe the default collections in Java and .NET should be supported out
of the box. Moreover, if we know the collection type, e.g. HashMap, we can
always provide a more efficient way of serializing it ourselves, in the
Binary marshaller.

Is this something you had in mind, or were you proposing something
different.

D.

On Tue, Dec 8, 2015 at 6:29 AM, Vladimir Ozerov <vo...@gridgain.com>
wrote:

> Alex,
>
> What interface do you mean? If user has collection field in class and
> explicitly call BinaryWriter.writeCollection(), we can leave current
> interoperability support - it is not a problem.
> As per your second point - user could pass collections e.g. as argument to
> Java task started from .NET. This is where we will loose interoperabiltiy
> and will force user to create some wrappers. But these are very specific
> use cases.
>
> BTW, proposed solution is almost exactly how we work with collections in
> .NET.
>
> On Tue, Dec 8, 2015 at 4:57 PM, Alexey Goncharuk <
> alexey.goncharuk@gmail.com
> > wrote:
>
> > I like the idea, however it has obvious downsides. First, if a user class
> > contains a collection, we force user to implement additional interface,
> > even if the collection is a simple ArrayList. Second, I do not see how
> this
> > plain collection can be the value for the cache - user will always need
> to
> > write a wrapper/containing class around it.
> >
> > I think we should provide minimum support for basic types - HashMap,
> > LinkedHashMap, ArrayList and treat other classes the way Vladimir
> > described.
> >
>

Re: Handling collections in BinaryMarshaller

[Vladimir Ozerov <vo...@gridgain.com>]

Alex,

What interface do you mean? If user has collection field in class and
explicitly call BinaryWriter.writeCollection(), we can leave current
interoperability support - it is not a problem.
As per your second point - user could pass collections e.g. as argument to
Java task started from .NET. This is where we will loose interoperabiltiy
and will force user to create some wrappers. But these are very specific
use cases.

BTW, proposed solution is almost exactly how we work with collections in
.NET.

On Tue, Dec 8, 2015 at 4:57 PM, Alexey Goncharuk <alexey.goncharuk@gmail.com
> wrote:

> I like the idea, however it has obvious downsides. First, if a user class
> contains a collection, we force user to implement additional interface,
> even if the collection is a simple ArrayList. Second, I do not see how this
> plain collection can be the value for the cache - user will always need to
> write a wrapper/containing class around it.
>
> I think we should provide minimum support for basic types - HashMap,
> LinkedHashMap, ArrayList and treat other classes the way Vladimir
> described.
>

Re: Handling collections in BinaryMarshaller

[Alexey Goncharuk <al...@gmail.com>]

I like the idea, however it has obvious downsides. First, if a user class
contains a collection, we force user to implement additional interface,
even if the collection is a simple ArrayList. Second, I do not see how this
plain collection can be the value for the cache - user will always need to
write a wrapper/containing class around it.

I think we should provide minimum support for basic types - HashMap,
LinkedHashMap, ArrayList and treat other classes the way Vladimir described.