You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Vyacheslav Daradur <da...@gmail.com> on 2018/12/07 12:56:52 UTC
Use of marshaller at node startup routine (need advice)
Igniters, I need your advice about the following problem:
It is necessary to serialize an object (just convert an object to
bytes array) for including it in joining node data (DiscoveryDataBag)
*at node startup routine*.
The marshalling hangs If we use 'BinaryMarshaller' or
'OptimizedMarshaler' because class can't be registered in
MarshallerContextImpl#registerClassName -> transport.proposeMapping on
account of the request which can't be sent through discovery-spi at
the moment.
Also, 'JdkMarshaller' can't be used because it imposes limits on
objects that should implement 'Serializable' interface. But this
restriction is unacceptable for the case.
As a workaround solution, an external library, like KRYO, can be used.
What tools also available in the project to solve this problem?
--
Best Regards, Vyacheslav D.
Re: Use of marshaller at node startup routine (need advice)
Posted by Vyacheslav Daradur <da...@gmail.com>.
In terms of providing consistency of services' API, where the
interface `Service` extend `Serializable` end-user should make his
implementation to be serializable properly.
From this point of view, `JdkMarshaller` can be used as the common
marshaller for service's instance, this allows reduce possible points
of pitfalls in comparison with the use of multiple marshallers.
The same approach is used in some other of Ignite's components, e.g.
in `TcpDiscocerySpi` (which is used for sending of service's change
state requests), `GridQueryProcessor` and others.
In the new implementation of the service processor, we use
`JdkMarshaller` as a common marshaller.
`IgniteServiceConfigVariationsFullApiTest` has been fixed to test
modes {SERIALIZABLE, CUSTOM_SERIALIZABLE, EXTERNALIZABLE}.
Thanks!
On Wed, Dec 12, 2018 at 4:30 PM Vyacheslav Daradur <da...@gmail.com> wrote:
>
> A bit more:
>
> I implemented a solution which regards a node connection state: if the
> node already joined to the cluster, then configured marshaller will be
> used, otherwise, JdkMarshaller will be used.
>
> This affects static configurations if a service can't be serialized
> this will be logged (covered by test), it will be quite transparent
> for users.
>
> In general, I'm also for using JdkMarshaller regardless of conditions.
>
> But, it would be not friendly for our users who already use
> 'Binarylizable' interface for the services.
>
> I think 'IgniteServiceConfigVariationsFullApiTest' should not be
> removed at the moment since we don't delete the previous
> implementation. But we can fix it if needed.
>
> As you said, we have 2 approaches:
> 1) Use `JdkMarshaller` since 'Service' interface extends
> `Serializable` (and fix all the tests)
> 2) Choose the marshaller depending on the node's connection state;
>
> Both approaches seem good to me, in the current PR second approach has
> been implemented.
>
> It would be great if other Ignite's experts said their opinion.
>
> On Wed, Dec 12, 2018 at 3:52 PM Vyacheslav Daradur <da...@gmail.com> wrote:
> >
> > Denis, thank you for the answer.
> >
> > I've already implemented the same approach. It works well, new and old
> > tests have passed.
> > On Wed, Dec 12, 2018 at 3:47 PM Denis Mekhanikov <dm...@gmail.com> wrote:
> > >
> > > Slava,
> > >
> > > Interface *Service *extends *Serializable.* So, all services are supposed
> > > to be serializable by the JdkMarshaller.
> > > Usage of *BinaryMarshaller* or *OptimizedMarshaller* makes sense only from
> > > performance point of view.
> > > But I don't think, that we should try too hard to minimize the performance
> > > impact of services serialization,
> > > since it doesn't happen too often.
> > >
> > > There are some tests like *IgniteServiceConfigVariationsFullApiTest*, that
> > > check, that services, which are not
> > > serializable, can be successfully deployed. I think, these tests should be
> > > removed.
> > > It's reasonable to require serializability, since all *Services* are marked
> > > as *Serializable*, as I already mentioned.
> > >
> > > Slava and I discussed a possibility to choose a marshaller, depending on
> > > the node state.
> > > If a node is already connected to the cluster, then it could use a binary
> > > marshaller,
> > > otherwise JDK marshaller could be used.
> > > I think, if we decide to do so, then it will complicate logic and confuse
> > > users.
> > > This problem exists only for static services. They are not different from
> > > dynamic ones,
> > > except for the way of configuration, and a moment of deployment.
> > > I don't see, why different constraints should be applied to them.
> > >
> > > So, I'm for using the *JdkMarshaller* regardless of the service type or a
> > > node state.
> > >
> > > Denis
> > >
> > > пт, 7 дек. 2018 г. в 15:57, Vyacheslav Daradur <da...@gmail.com>:
> > >
> > > > Igniters, I need your advice about the following problem:
> > > >
> > > > It is necessary to serialize an object (just convert an object to
> > > > bytes array) for including it in joining node data (DiscoveryDataBag)
> > > > *at node startup routine*.
> > > >
> > > > The marshalling hangs If we use 'BinaryMarshaller' or
> > > > 'OptimizedMarshaler' because class can't be registered in
> > > > MarshallerContextImpl#registerClassName -> transport.proposeMapping on
> > > > account of the request which can't be sent through discovery-spi at
> > > > the moment.
> > > >
> > > > Also, 'JdkMarshaller' can't be used because it imposes limits on
> > > > objects that should implement 'Serializable' interface. But this
> > > > restriction is unacceptable for the case.
> > > >
> > > > As a workaround solution, an external library, like KRYO, can be used.
> > > >
> > > > What tools also available in the project to solve this problem?
> > > >
> > > > --
> > > > Best Regards, Vyacheslav D.
> > > >
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
>
>
>
> --
> Best Regards, Vyacheslav D.
--
Best Regards, Vyacheslav D.
Re: Use of marshaller at node startup routine (need advice)
Posted by Vyacheslav Daradur <da...@gmail.com>.
A bit more:
I implemented a solution which regards a node connection state: if the
node already joined to the cluster, then configured marshaller will be
used, otherwise, JdkMarshaller will be used.
This affects static configurations if a service can't be serialized
this will be logged (covered by test), it will be quite transparent
for users.
In general, I'm also for using JdkMarshaller regardless of conditions.
But, it would be not friendly for our users who already use
'Binarylizable' interface for the services.
I think 'IgniteServiceConfigVariationsFullApiTest' should not be
removed at the moment since we don't delete the previous
implementation. But we can fix it if needed.
As you said, we have 2 approaches:
1) Use `JdkMarshaller` since 'Service' interface extends
`Serializable` (and fix all the tests)
2) Choose the marshaller depending on the node's connection state;
Both approaches seem good to me, in the current PR second approach has
been implemented.
It would be great if other Ignite's experts said their opinion.
On Wed, Dec 12, 2018 at 3:52 PM Vyacheslav Daradur <da...@gmail.com> wrote:
>
> Denis, thank you for the answer.
>
> I've already implemented the same approach. It works well, new and old
> tests have passed.
> On Wed, Dec 12, 2018 at 3:47 PM Denis Mekhanikov <dm...@gmail.com> wrote:
> >
> > Slava,
> >
> > Interface *Service *extends *Serializable.* So, all services are supposed
> > to be serializable by the JdkMarshaller.
> > Usage of *BinaryMarshaller* or *OptimizedMarshaller* makes sense only from
> > performance point of view.
> > But I don't think, that we should try too hard to minimize the performance
> > impact of services serialization,
> > since it doesn't happen too often.
> >
> > There are some tests like *IgniteServiceConfigVariationsFullApiTest*, that
> > check, that services, which are not
> > serializable, can be successfully deployed. I think, these tests should be
> > removed.
> > It's reasonable to require serializability, since all *Services* are marked
> > as *Serializable*, as I already mentioned.
> >
> > Slava and I discussed a possibility to choose a marshaller, depending on
> > the node state.
> > If a node is already connected to the cluster, then it could use a binary
> > marshaller,
> > otherwise JDK marshaller could be used.
> > I think, if we decide to do so, then it will complicate logic and confuse
> > users.
> > This problem exists only for static services. They are not different from
> > dynamic ones,
> > except for the way of configuration, and a moment of deployment.
> > I don't see, why different constraints should be applied to them.
> >
> > So, I'm for using the *JdkMarshaller* regardless of the service type or a
> > node state.
> >
> > Denis
> >
> > пт, 7 дек. 2018 г. в 15:57, Vyacheslav Daradur <da...@gmail.com>:
> >
> > > Igniters, I need your advice about the following problem:
> > >
> > > It is necessary to serialize an object (just convert an object to
> > > bytes array) for including it in joining node data (DiscoveryDataBag)
> > > *at node startup routine*.
> > >
> > > The marshalling hangs If we use 'BinaryMarshaller' or
> > > 'OptimizedMarshaler' because class can't be registered in
> > > MarshallerContextImpl#registerClassName -> transport.proposeMapping on
> > > account of the request which can't be sent through discovery-spi at
> > > the moment.
> > >
> > > Also, 'JdkMarshaller' can't be used because it imposes limits on
> > > objects that should implement 'Serializable' interface. But this
> > > restriction is unacceptable for the case.
> > >
> > > As a workaround solution, an external library, like KRYO, can be used.
> > >
> > > What tools also available in the project to solve this problem?
> > >
> > > --
> > > Best Regards, Vyacheslav D.
> > >
>
>
>
> --
> Best Regards, Vyacheslav D.
--
Best Regards, Vyacheslav D.
Re: Use of marshaller at node startup routine (need advice)
Posted by Vyacheslav Daradur <da...@gmail.com>.
Denis, thank you for the answer.
I've already implemented the same approach. It works well, new and old
tests have passed.
On Wed, Dec 12, 2018 at 3:47 PM Denis Mekhanikov <dm...@gmail.com> wrote:
>
> Slava,
>
> Interface *Service *extends *Serializable.* So, all services are supposed
> to be serializable by the JdkMarshaller.
> Usage of *BinaryMarshaller* or *OptimizedMarshaller* makes sense only from
> performance point of view.
> But I don't think, that we should try too hard to minimize the performance
> impact of services serialization,
> since it doesn't happen too often.
>
> There are some tests like *IgniteServiceConfigVariationsFullApiTest*, that
> check, that services, which are not
> serializable, can be successfully deployed. I think, these tests should be
> removed.
> It's reasonable to require serializability, since all *Services* are marked
> as *Serializable*, as I already mentioned.
>
> Slava and I discussed a possibility to choose a marshaller, depending on
> the node state.
> If a node is already connected to the cluster, then it could use a binary
> marshaller,
> otherwise JDK marshaller could be used.
> I think, if we decide to do so, then it will complicate logic and confuse
> users.
> This problem exists only for static services. They are not different from
> dynamic ones,
> except for the way of configuration, and a moment of deployment.
> I don't see, why different constraints should be applied to them.
>
> So, I'm for using the *JdkMarshaller* regardless of the service type or a
> node state.
>
> Denis
>
> пт, 7 дек. 2018 г. в 15:57, Vyacheslav Daradur <da...@gmail.com>:
>
> > Igniters, I need your advice about the following problem:
> >
> > It is necessary to serialize an object (just convert an object to
> > bytes array) for including it in joining node data (DiscoveryDataBag)
> > *at node startup routine*.
> >
> > The marshalling hangs If we use 'BinaryMarshaller' or
> > 'OptimizedMarshaler' because class can't be registered in
> > MarshallerContextImpl#registerClassName -> transport.proposeMapping on
> > account of the request which can't be sent through discovery-spi at
> > the moment.
> >
> > Also, 'JdkMarshaller' can't be used because it imposes limits on
> > objects that should implement 'Serializable' interface. But this
> > restriction is unacceptable for the case.
> >
> > As a workaround solution, an external library, like KRYO, can be used.
> >
> > What tools also available in the project to solve this problem?
> >
> > --
> > Best Regards, Vyacheslav D.
> >
--
Best Regards, Vyacheslav D.
Re: Use of marshaller at node startup routine (need advice)
Posted by Denis Mekhanikov <dm...@gmail.com>.
Slava,
Interface *Service *extends *Serializable.* So, all services are supposed
to be serializable by the JdkMarshaller.
Usage of *BinaryMarshaller* or *OptimizedMarshaller* makes sense only from
performance point of view.
But I don't think, that we should try too hard to minimize the performance
impact of services serialization,
since it doesn't happen too often.
There are some tests like *IgniteServiceConfigVariationsFullApiTest*, that
check, that services, which are not
serializable, can be successfully deployed. I think, these tests should be
removed.
It's reasonable to require serializability, since all *Services* are marked
as *Serializable*, as I already mentioned.
Slava and I discussed a possibility to choose a marshaller, depending on
the node state.
If a node is already connected to the cluster, then it could use a binary
marshaller,
otherwise JDK marshaller could be used.
I think, if we decide to do so, then it will complicate logic and confuse
users.
This problem exists only for static services. They are not different from
dynamic ones,
except for the way of configuration, and a moment of deployment.
I don't see, why different constraints should be applied to them.
So, I'm for using the *JdkMarshaller* regardless of the service type or a
node state.
Denis
пт, 7 дек. 2018 г. в 15:57, Vyacheslav Daradur <da...@gmail.com>:
> Igniters, I need your advice about the following problem:
>
> It is necessary to serialize an object (just convert an object to
> bytes array) for including it in joining node data (DiscoveryDataBag)
> *at node startup routine*.
>
> The marshalling hangs If we use 'BinaryMarshaller' or
> 'OptimizedMarshaler' because class can't be registered in
> MarshallerContextImpl#registerClassName -> transport.proposeMapping on
> account of the request which can't be sent through discovery-spi at
> the moment.
>
> Also, 'JdkMarshaller' can't be used because it imposes limits on
> objects that should implement 'Serializable' interface. But this
> restriction is unacceptable for the case.
>
> As a workaround solution, an external library, like KRYO, can be used.
>
> What tools also available in the project to solve this problem?
>
> --
> Best Regards, Vyacheslav D.
>