You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Valentin Kulichenko <va...@gmail.com> on 2016/02/20 04:15:07 UTC
Binary object inside Externalizable
Folks,
I recently faced an issue which seems to be pretty critical. As you know,
when binary marshaller meets an Externalizable object, it switches to
optimized marshaller. And if there is a regular object inside, it's still
serialized with optimized, even if its type is declared in binary
configuration with custom mapper, etc. This looks wrong.
It's getting even worse in compute grid, because closure processor wraps
user's closures in some internal classes, which are Externalizable, so
these closures are always serialized with optimized marshaller.
Does anyone have ideas on what is the proper fix for this?
-Val
Re: Binary object inside Externalizable
Posted by Dmitriy Setrakyan <ds...@apache.org>.
This would be a difficult issue to solve because of circular references.
Can we in the mean time just make sure that none of Ignite internal objects
are externalizable, so we don’t purposely force this situation onto our
users?
D.
On Fri, Feb 19, 2016 at 7:15 PM, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Folks,
>
> I recently faced an issue which seems to be pretty critical. As you know,
> when binary marshaller meets an Externalizable object, it switches to
> optimized marshaller. And if there is a regular object inside, it's still
> serialized with optimized, even if its type is declared in binary
> configuration with custom mapper, etc. This looks wrong.
>
> It's getting even worse in compute grid, because closure processor wraps
> user's closures in some internal classes, which are Externalizable, so
> these closures are always serialized with optimized marshaller.
>
> Does anyone have ideas on what is the proper fix for this?
>
> -Val
>
Re: Binary object inside Externalizable
Posted by Dmitriy Setrakyan <ds...@apache.org>.
I am always in favor of easy fixes :)
On Tue, Feb 23, 2016 at 8:50 PM, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Actually, adding Externalizable support to binary marshaller should not be
> difficult, no? BinaryWriterExImpl already implements ObjectOutput, so we
> just need to pass to writeExternal (and the same for deserialization). To
> be honest, I don't understand why we use optimized marshaller here and
> making this change should fix majority of use cases.
>
> The issue will remain for writeObject/readObject which is much more
> complicated and where using optimized marshaller makes more sense. But this
> is a relatively rare case and I think we can fix this as a next step.
>
> Thoughts?
>
> On Sat, Feb 20, 2016 at 10:01 AM, Dmitriy Setrakyan <dsetrakyan@apache.org
> >
> wrote:
>
> > Andrey, you are absolutely right. I was suggesting a quick fix, which in
> my
> > view, would fix most of the issues. In the long term, we should fix the
> > binary serialization to work the way you are describing.
> >
> > D.
> >
> > On Sat, Feb 20, 2016 at 9:26 AM, Andrey Kornev <andrewkornev@hotmail.com
> >
> > wrote:
> >
> > > Val,
> > >
> > > I think Ignite Serialization should follow the approach similar to what
> > > Java serialization does: it can transparently and consistently handle
> > both
> > > Serializable and Externalizable classes. It can do it because it relies
> > on
> > > a single class - ObjectOutputStream - to do serialization (and a single
> > > class - ObjectInputStream - to do the opposite, but let's ignore it for
> > the
> > > moment). ObjectOutputStream delegates to proper user callbacks
> > (writeObject
> > > or writeExternal) at the right moments, but otherwise it fully controls
> > the
> > > on-the-wire representation and takes care of all the necessary stream
> > > framing/marking so that ObjectInputStream can then correctly
> deserialize
> > > the bytes. It's never a problem for Java to serialize an Externalizable
> > > field from a Serializable object and vice versa. Users can mix and
> match
> > > serialization APIs as they please.
> > >
> > > I think Ignite should just have a single marshaller, let's say the
> > Binary,
> > > which drives the whole serialization process. It doesn't delegate to
> > > OptimizedMarshaller as it does today. Instead it detects the
> > > Serializable/Externalizable classes and calls their serialization
> > callbacks
> > > - writeObject/writeExternal - passing in *itself* as
> > > ObjectOutputStream/ObjectOutput correspondingly. This way the entire
> > > serialization process never leaves the context of a single Binary
> > > marshaller and allows Ignite to handle such complex cases as the one
> > we're
> > > discussing now.
> > >
> > > I hope it makes sense.
> > >
> > > Andrey
> > >
> > > > From: valentin.kulichenko@gmail.com
> > > > Date: Fri, 19 Feb 2016 19:15:07 -0800
> > > > Subject: Binary object inside Externalizable
> > > > To: dev@ignite.apache.org
> > > >
> > > > Folks,
> > > >
> > > > I recently faced an issue which seems to be pretty critical. As you
> > know,
> > > > when binary marshaller meets an Externalizable object, it switches to
> > > > optimized marshaller. And if there is a regular object inside, it's
> > still
> > > > serialized with optimized, even if its type is declared in binary
> > > > configuration with custom mapper, etc. This looks wrong.
> > > >
> > > > It's getting even worse in compute grid, because closure processor
> > wraps
> > > > user's closures in some internal classes, which are Externalizable,
> so
> > > > these closures are always serialized with optimized marshaller.
> > > >
> > > > Does anyone have ideas on what is the proper fix for this?
> > > >
> > > > -Val
> > >
> > >
> >
>
RE: Binary object inside Externalizable
Posted by Andrey Kornev <an...@hotmail.com>.
Val,
In our case, it's other way around -- use of Externalizable is pretty rare and it's mostly Serializables throughout the code base (not our code, but of our clients that use our software). Besides there are always those pesky 3d party libraries and the JDK itself (for example, the Throwable class) that use Serializable.
My point is that no matter how much we all love quick fixes, when it comes to serialization, all cases must be properly supported.
Regards
Andrey
> From: valentin.kulichenko@gmail.com
> Date: Tue, 23 Feb 2016 20:50:14 -0800
> Subject: Re: Binary object inside Externalizable
> To: dev@ignite.apache.org
>
> Actually, adding Externalizable support to binary marshaller should not be
> difficult, no? BinaryWriterExImpl already implements ObjectOutput, so we
> just need to pass to writeExternal (and the same for deserialization). To
> be honest, I don't understand why we use optimized marshaller here and
> making this change should fix majority of use cases.
>
> The issue will remain for writeObject/readObject which is much more
> complicated and where using optimized marshaller makes more sense. But this
> is a relatively rare case and I think we can fix this as a next step.
>
> Thoughts?
>
> On Sat, Feb 20, 2016 at 10:01 AM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > Andrey, you are absolutely right. I was suggesting a quick fix, which in my
> > view, would fix most of the issues. In the long term, we should fix the
> > binary serialization to work the way you are describing.
> >
> > D.
> >
> > On Sat, Feb 20, 2016 at 9:26 AM, Andrey Kornev <an...@hotmail.com>
> > wrote:
> >
> > > Val,
> > >
> > > I think Ignite Serialization should follow the approach similar to what
> > > Java serialization does: it can transparently and consistently handle
> > both
> > > Serializable and Externalizable classes. It can do it because it relies
> > on
> > > a single class - ObjectOutputStream - to do serialization (and a single
> > > class - ObjectInputStream - to do the opposite, but let's ignore it for
> > the
> > > moment). ObjectOutputStream delegates to proper user callbacks
> > (writeObject
> > > or writeExternal) at the right moments, but otherwise it fully controls
> > the
> > > on-the-wire representation and takes care of all the necessary stream
> > > framing/marking so that ObjectInputStream can then correctly deserialize
> > > the bytes. It's never a problem for Java to serialize an Externalizable
> > > field from a Serializable object and vice versa. Users can mix and match
> > > serialization APIs as they please.
> > >
> > > I think Ignite should just have a single marshaller, let's say the
> > Binary,
> > > which drives the whole serialization process. It doesn't delegate to
> > > OptimizedMarshaller as it does today. Instead it detects the
> > > Serializable/Externalizable classes and calls their serialization
> > callbacks
> > > - writeObject/writeExternal - passing in *itself* as
> > > ObjectOutputStream/ObjectOutput correspondingly. This way the entire
> > > serialization process never leaves the context of a single Binary
> > > marshaller and allows Ignite to handle such complex cases as the one
> > we're
> > > discussing now.
> > >
> > > I hope it makes sense.
> > >
> > > Andrey
> > >
> > > > From: valentin.kulichenko@gmail.com
> > > > Date: Fri, 19 Feb 2016 19:15:07 -0800
> > > > Subject: Binary object inside Externalizable
> > > > To: dev@ignite.apache.org
> > > >
> > > > Folks,
> > > >
> > > > I recently faced an issue which seems to be pretty critical. As you
> > know,
> > > > when binary marshaller meets an Externalizable object, it switches to
> > > > optimized marshaller. And if there is a regular object inside, it's
> > still
> > > > serialized with optimized, even if its type is declared in binary
> > > > configuration with custom mapper, etc. This looks wrong.
> > > >
> > > > It's getting even worse in compute grid, because closure processor
> > wraps
> > > > user's closures in some internal classes, which are Externalizable, so
> > > > these closures are always serialized with optimized marshaller.
> > > >
> > > > Does anyone have ideas on what is the proper fix for this?
> > > >
> > > > -Val
> > >
> > >
> >
Re: Binary object inside Externalizable
Posted by Valentin Kulichenko <va...@gmail.com>.
Actually, adding Externalizable support to binary marshaller should not be
difficult, no? BinaryWriterExImpl already implements ObjectOutput, so we
just need to pass to writeExternal (and the same for deserialization). To
be honest, I don't understand why we use optimized marshaller here and
making this change should fix majority of use cases.
The issue will remain for writeObject/readObject which is much more
complicated and where using optimized marshaller makes more sense. But this
is a relatively rare case and I think we can fix this as a next step.
Thoughts?
On Sat, Feb 20, 2016 at 10:01 AM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> Andrey, you are absolutely right. I was suggesting a quick fix, which in my
> view, would fix most of the issues. In the long term, we should fix the
> binary serialization to work the way you are describing.
>
> D.
>
> On Sat, Feb 20, 2016 at 9:26 AM, Andrey Kornev <an...@hotmail.com>
> wrote:
>
> > Val,
> >
> > I think Ignite Serialization should follow the approach similar to what
> > Java serialization does: it can transparently and consistently handle
> both
> > Serializable and Externalizable classes. It can do it because it relies
> on
> > a single class - ObjectOutputStream - to do serialization (and a single
> > class - ObjectInputStream - to do the opposite, but let's ignore it for
> the
> > moment). ObjectOutputStream delegates to proper user callbacks
> (writeObject
> > or writeExternal) at the right moments, but otherwise it fully controls
> the
> > on-the-wire representation and takes care of all the necessary stream
> > framing/marking so that ObjectInputStream can then correctly deserialize
> > the bytes. It's never a problem for Java to serialize an Externalizable
> > field from a Serializable object and vice versa. Users can mix and match
> > serialization APIs as they please.
> >
> > I think Ignite should just have a single marshaller, let's say the
> Binary,
> > which drives the whole serialization process. It doesn't delegate to
> > OptimizedMarshaller as it does today. Instead it detects the
> > Serializable/Externalizable classes and calls their serialization
> callbacks
> > - writeObject/writeExternal - passing in *itself* as
> > ObjectOutputStream/ObjectOutput correspondingly. This way the entire
> > serialization process never leaves the context of a single Binary
> > marshaller and allows Ignite to handle such complex cases as the one
> we're
> > discussing now.
> >
> > I hope it makes sense.
> >
> > Andrey
> >
> > > From: valentin.kulichenko@gmail.com
> > > Date: Fri, 19 Feb 2016 19:15:07 -0800
> > > Subject: Binary object inside Externalizable
> > > To: dev@ignite.apache.org
> > >
> > > Folks,
> > >
> > > I recently faced an issue which seems to be pretty critical. As you
> know,
> > > when binary marshaller meets an Externalizable object, it switches to
> > > optimized marshaller. And if there is a regular object inside, it's
> still
> > > serialized with optimized, even if its type is declared in binary
> > > configuration with custom mapper, etc. This looks wrong.
> > >
> > > It's getting even worse in compute grid, because closure processor
> wraps
> > > user's closures in some internal classes, which are Externalizable, so
> > > these closures are always serialized with optimized marshaller.
> > >
> > > Does anyone have ideas on what is the proper fix for this?
> > >
> > > -Val
> >
> >
>
Re: Binary object inside Externalizable
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Andrey, you are absolutely right. I was suggesting a quick fix, which in my
view, would fix most of the issues. In the long term, we should fix the
binary serialization to work the way you are describing.
D.
On Sat, Feb 20, 2016 at 9:26 AM, Andrey Kornev <an...@hotmail.com>
wrote:
> Val,
>
> I think Ignite Serialization should follow the approach similar to what
> Java serialization does: it can transparently and consistently handle both
> Serializable and Externalizable classes. It can do it because it relies on
> a single class - ObjectOutputStream - to do serialization (and a single
> class - ObjectInputStream - to do the opposite, but let's ignore it for the
> moment). ObjectOutputStream delegates to proper user callbacks (writeObject
> or writeExternal) at the right moments, but otherwise it fully controls the
> on-the-wire representation and takes care of all the necessary stream
> framing/marking so that ObjectInputStream can then correctly deserialize
> the bytes. It's never a problem for Java to serialize an Externalizable
> field from a Serializable object and vice versa. Users can mix and match
> serialization APIs as they please.
>
> I think Ignite should just have a single marshaller, let's say the Binary,
> which drives the whole serialization process. It doesn't delegate to
> OptimizedMarshaller as it does today. Instead it detects the
> Serializable/Externalizable classes and calls their serialization callbacks
> - writeObject/writeExternal - passing in *itself* as
> ObjectOutputStream/ObjectOutput correspondingly. This way the entire
> serialization process never leaves the context of a single Binary
> marshaller and allows Ignite to handle such complex cases as the one we're
> discussing now.
>
> I hope it makes sense.
>
> Andrey
>
> > From: valentin.kulichenko@gmail.com
> > Date: Fri, 19 Feb 2016 19:15:07 -0800
> > Subject: Binary object inside Externalizable
> > To: dev@ignite.apache.org
> >
> > Folks,
> >
> > I recently faced an issue which seems to be pretty critical. As you know,
> > when binary marshaller meets an Externalizable object, it switches to
> > optimized marshaller. And if there is a regular object inside, it's still
> > serialized with optimized, even if its type is declared in binary
> > configuration with custom mapper, etc. This looks wrong.
> >
> > It's getting even worse in compute grid, because closure processor wraps
> > user's closures in some internal classes, which are Externalizable, so
> > these closures are always serialized with optimized marshaller.
> >
> > Does anyone have ideas on what is the proper fix for this?
> >
> > -Val
>
>
RE: Binary object inside Externalizable
Posted by Andrey Kornev <an...@hotmail.com>.
Val,
I think Ignite Serialization should follow the approach similar to what Java serialization does: it can transparently and consistently handle both Serializable and Externalizable classes. It can do it because it relies on a single class - ObjectOutputStream - to do serialization (and a single class - ObjectInputStream - to do the opposite, but let's ignore it for the moment). ObjectOutputStream delegates to proper user callbacks (writeObject or writeExternal) at the right moments, but otherwise it fully controls the on-the-wire representation and takes care of all the necessary stream framing/marking so that ObjectInputStream can then correctly deserialize the bytes. It's never a problem for Java to serialize an Externalizable field from a Serializable object and vice versa. Users can mix and match serialization APIs as they please.
I think Ignite should just have a single marshaller, let's say the Binary, which drives the whole serialization process. It doesn't delegate to OptimizedMarshaller as it does today. Instead it detects the Serializable/Externalizable classes and calls their serialization callbacks - writeObject/writeExternal - passing in *itself* as ObjectOutputStream/ObjectOutput correspondingly. This way the entire serialization process never leaves the context of a single Binary marshaller and allows Ignite to handle such complex cases as the one we're discussing now.
I hope it makes sense.
Andrey
> From: valentin.kulichenko@gmail.com
> Date: Fri, 19 Feb 2016 19:15:07 -0800
> Subject: Binary object inside Externalizable
> To: dev@ignite.apache.org
>
> Folks,
>
> I recently faced an issue which seems to be pretty critical. As you know,
> when binary marshaller meets an Externalizable object, it switches to
> optimized marshaller. And if there is a regular object inside, it's still
> serialized with optimized, even if its type is declared in binary
> configuration with custom mapper, etc. This looks wrong.
>
> It's getting even worse in compute grid, because closure processor wraps
> user's closures in some internal classes, which are Externalizable, so
> these closures are always serialized with optimized marshaller.
>
> Does anyone have ideas on what is the proper fix for this?
>
> -Val