You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by "Gilles, Romain" <ro...@misys.com> on 2015/11/02 10:56:53 UTC
OSGi migration may require a special marshaller
Hi all,
I'm really interested in this issue: https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to make it work in our osgi environment. The main issue for us now is the serialization. I think it you will have to rework the OptimizedMarshaller or any other marshaller that works with object that come from outside your class space.
We have try kryo that works. Kryo provide an extension point in order to resolve the classes: https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java . With this extension we are able to solve the problem of external classes. The only issue with kryo is that some classes need a certain care in the serialization process and therefore a specialized serializer.
So I would like to know from the community what do think of changing the way the optimized marshaller works or introducing the support of yet another marshaller based on a kryo like technology?
Thanks in advance,
Best regards,
Romain.
PS: I'm ready to help in the both cases.
"Misys" is the trade name of the Misys group of companies. This email and any attachments have been scanned for known viruses using multiple scanners. This email message is intended for the named recipient only. It may be privileged and/or confidential. If you are not the named recipient of this email please notify us immediately and do not copy it or use it for any purpose, nor disclose its contents to any other person. This email does not constitute the commencement of legal relations between you and Misys. Please refer to the executed contract between you and the relevant member of the Misys group for the identity of the contracting party with which you are dealing.
Re: OSGi migration may require a special marshaller
Posted by Yakov Zhdanov <yz...@gridgain.com>.
Romain,
Is that correct that you obtain instance of Ignite through cache manager
and pass your classloader to which gets ignored at some point? Or you
use IgniteConfiguration#setClassLoader?
Can you or anyone in community please provide a short example or junit so I
can reproduce this?
Thanks!
Thanks!
--
Yakov Zhdanov, Director R&D
*GridGain Systems*
www.gridgain.com
2015-11-02 12:56 GMT+03:00 Gilles, Romain <ro...@misys.com>:
> Hi all,
>
> I'm really interested in this issue:
> https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to make
> it work in our osgi environment. The main issue for us now is the
> serialization. I think it you will have to rework the OptimizedMarshaller
> or any other marshaller that works with object that come from outside your
> class space.
>
> We have try kryo that works. Kryo provide an extension point in order to
> resolve the classes:
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> . With this extension we are able to solve the problem of external classes.
> The only issue with kryo is that some classes need a certain care in the
> serialization process and therefore a specialized serializer.
>
> So I would like to know from the community what do think of changing the
> way the optimized marshaller works or introducing the support of yet
> another marshaller based on a kryo like technology?
>
>
> Thanks in advance,
>
>
> Best regards,
>
>
> Romain.
>
>
> PS: I'm ready to help in the both cases.
> "Misys" is the trade name of the Misys group of companies. This email and
> any attachments have been scanned for known viruses using multiple
> scanners. This email message is intended for the named recipient only. It
> may be privileged and/or confidential. If you are not the named recipient
> of this email please notify us immediately and do not copy it or use it for
> any purpose, nor disclose its contents to any other person. This email does
> not constitute the commencement of legal relations between you and Misys.
> Please refer to the executed contract between you and the relevant member
> of the Misys group for the identity of the contracting party with which you
> are dealing.
>
Re: OSGi migration may require a special marshaller
Posted by Romain Gilles <ro...@gmail.com>.
Hi Raul,
I had a look to what has been done in camel. I think here the use cases
came be different.
Let say I want to run a closure against the grid like this:
IgniteCompute compute = ignite.compute();
// Execute closure on all cluster nodes.Collection<Integer> res = compute.apply(
String::length,
Arrays.asList("How many characters".split(" "))
);
// Add all the word lengths received from cluster nodes.
int total = res.stream().mapToInt(Integer::intValue).sum();
Then I think, even in more complex and realistic use cases, you may don't
want to export the implementation details of your closure. But this closure
will be serialized by the Ignite marshalling framework.
Romain.
Le mar. 3 nov. 2015 à 20:18, Raul Kripalani <ra...@apache.org> a écrit :
> Hey guys,
>
> About the TCCL, I need to dig deeper into how serialisation is being done
> now. If, at any point, we are resolving a class through the classloader of
> a particular class, e.g. Ignition.getClass().getClassLoader(), etc. we are
> using the class space of the bundle containing that class. If we are using
> a class from Ignite, we obviously wouldn't be able to find a custom DTO
> provided by a user.
>
> In Camel we have found this problem several times and we've solved it by
> changing the TCCL, or using the TCCL [1] [2] to resolve classes.
>
> As I said, I need to look into this deeper and I'll have some time on
> Thursday, so I hope you don't mind waiting a little bit. I have already
> "bundle-fied" most of Ignite modules in the corresponding branch [3], so my
> next step is to test out the Ignite examples inside an OSGi environment
> (Karaf 4).
>
> [1] https://issues.apache.org/jira/browse/CAMEL-5748
> [2] https://issues.apache.org/jira/browse/CAMEL-5722
> [3] https://github.com/apache/ignite/tree/ignite-1527
>
> Regards,
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>
> On Tue, Nov 3, 2015 at 10:38 AM, Romain Gilles <ro...@gmail.com>
> wrote:
>
> > Hi Raul,
> > - Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
> > - I like the point 2. Maybe for a graph of object that come from
> different
> > packages / bundles you may have to recursively capture the package
> version
> > of the individual element of your graph.
> > - For the point 3 I wonder if it will not over-complexify the solution.
> As
> > a developer you will have to think about it. And it is not obvious in the
> > code where things are serialized or not. You may use lambda in your
> > application code therefore the current package become what you call a DTO
> > package. The current state of ignite does not enforce you to specify it
> for
> > "classical" classloading application. If you introduce this extra step
> for
> > OSGi ready application you will maybe discourage people to use OSGi.
> >
> > To comeback to our solution. We start we a strong assumption: our cluster
> > is homogeneous in term of code so, of course, it simplify our life :).
> >
> > BTW if you can introduce an extension point in the class resolution
> > mechanism it can be interesting for end user in order to allow them to
> > optimize it based on there specific use cases.
> >
> > Romain.
> >
> > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <ra...@evosent.com> a écrit :
> >
> > > Hi Andrey,
> > >
> > > Thanks for the participation in this topic.
> > >
> > > I don't like the solution to incorporate the bundle symbolic name in
> the
> > > serialised form. Nothing guarantees that the classdef will be located
> > under
> > > the same bundle in both source and target machines. We also have to
> take
> > > into account that OSGi allows for multiple versions of the same
> > > bundle/packages to co-exist in the same container. So it becomes more
> > > complex.
> > >
> > > Using the TCCL should work when serialising, but it'll probably be of
> no
> > > use when deserialising on the other end.
> > >
> > > I need to enhance Ignite to:
> > >
> > > 1. Use the TCCL when marshalling on one end.
> > > 2. Incorporate the package version of the class in the serialised form
> if
> > > Ignite is running in an OSGi environment.
> > > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > > through a custom OSGi manifest header or the like, as I explained
> before.
> > > Package version must be taken into account.
> > >
> > > What do you think?
> > >
> > > Raúl.
> > > On 2 Nov 2015 17:25, "Andrey Kornev" <an...@hotmail.com> wrote:
> > >
> > > > Raul,
> > > >
> > > > The fundamental hurdle we need to jump over to make Ignite
> OSGi-enabled
> > > is
> > > > the marshalling. More specifically the issue is with deserialization
> of
> > > the
> > > > classes that are provided by the bundles other than the Ignite bundle
> > > > itself.
> > > >
> > > > When the Ignite transport layer receives a message it needs to figure
> > out
> > > > how to deserialize the bytes and for that it needs to know the bundle
> > > that
> > > > provides the class to be deserailized. At this point TCCL is of no
> use.
> > > To
> > > > make things more complex, the class may contain other classes that
> come
> > > > from other bundles, and so on recursively. This means that each
> object
> > in
> > > > the hierarchy must be serialized with its bundle name (or bundle id),
> > so
> > > > that the deserializer will then be able to correctly resolve the
> class
> > > > while traversing the object hierarchy during deserialization.
> > > >
> > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> > > plug
> > > > in a custom class resolver. Romain's solution was to use Kryo that
> does
> > > > provide a way to customize class resolution. It has solved the
> problem
> > of
> > > > capturing the bundle info and he was able to successfully run Ignite
> > as a
> > > > bundle in an OSGi container (after some repackaging and inclusion of
> > the
> > > > manifest). But Kryo-based marshalling introduced a lot of complexity
> to
> > > the
> > > > code and incorrect use of Kryo's numerous serializers caused some
> weird
> > > > hard-to-debug issues in the Ignite core (like duplicate cache entries
> > due
> > > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > > > figure!). Overall the Kryo-based solution is brittle and hard to
> > > maintain.
> > > >
> > > > I feel the correct solution to OSGi problem would be to
> > > > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > > > original one) to be used in OSGi environment.
> > > >
> > > > Regards
> > > > Andrey
> > > >
> > > > > From: raulk@apache.org
> > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > To: dev@ignite.apache.org
> > > > >
> > > > > Hi Romain,
> > > > >
> > > > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> > > input.
> > > > >
> > > > > I'm thinking about the situation you describe and I wonder if
> you're
> > > > > exporting Ignite as an OSGi service which is then consumed from
> other
> > > > > bundles. Under this situation, it would be quite easy to reproduce
> > the
> > > > > behaviour you describe if Ignite is not resolving classes via the
> > TCCL.
> > > > > Need to dig deeper into that.
> > > > >
> > > > > Off the top of my head, there are two alternatives to solve it:
> > > > >
> > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already
> used) –
> > > we
> > > > > gotta be wary of possible regressions.
> > > > > 2. Create a special OSGi header Ignite-Export-Package so that
> bundles
> > > > > containing DTOs can expose packages to Ignite's marshallers.
> > > > >
> > > > > Regards,
> > > > >
> > > > > *Raúl Kripalani*
> > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> Data
> > > and
> > > > > Messaging Engineer
> > > > > http://about.me/raulkripalani |
> > > http://www.linkedin.com/in/raulkripalani
> > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > >
> > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > romain.gilles@misys.com>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'm really interested in this issue:
> > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some
> stuff
> > to
> > > > make
> > > > > > it work in our osgi environment. The main issue for us now is the
> > > > > > serialization. I think it you will have to rework the
> > > > OptimizedMarshaller
> > > > > > or any other marshaller that works with object that come from
> > outside
> > > > your
> > > > > > class space.
> > > > > >
> > > > > > We have try kryo that works. Kryo provide an extension point in
> > order
> > > > to
> > > > > > resolve the classes:
> > > > > >
> > > >
> > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > . With this extension we are able to solve the problem of
> external
> > > > classes.
> > > > > > The only issue with kryo is that some classes need a certain care
> > in
> > > > the
> > > > > > serialization process and therefore a specialized serializer.
> > > > > >
> > > > > > So I would like to know from the community what do think of
> > changing
> > > > the
> > > > > > way the optimized marshaller works or introducing the support of
> > yet
> > > > > > another marshaller based on a kryo like technology?
> > > > > >
> > > > > >
> > > > > > Thanks in advance,
> > > > > >
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > >
> > > > > > Romain.
> > > > > >
> > > > > >
> > > > > > PS: I'm ready to help in the both cases.
> > > > > > "Misys" is the trade name of the Misys group of companies. This
> > email
> > > > and
> > > > > > any attachments have been scanned for known viruses using
> multiple
> > > > > > scanners. This email message is intended for the named recipient
> > > only.
> > > > It
> > > > > > may be privileged and/or confidential. If you are not the named
> > > > recipient
> > > > > > of this email please notify us immediately and do not copy it or
> > use
> > > > it for
> > > > > > any purpose, nor disclose its contents to any other person. This
> > > email
> > > > does
> > > > > > not constitute the commencement of legal relations between you
> and
> > > > Misys.
> > > > > > Please refer to the executed contract between you and the
> relevant
> > > > member
> > > > > > of the Misys group for the identity of the contracting party with
> > > > which you
> > > > > > are dealing.
> > > > > >
> > > >
> > >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Raul Kripalani <ra...@apache.org>.
Hey guys,
About the TCCL, I need to dig deeper into how serialisation is being done
now. If, at any point, we are resolving a class through the classloader of
a particular class, e.g. Ignition.getClass().getClassLoader(), etc. we are
using the class space of the bundle containing that class. If we are using
a class from Ignite, we obviously wouldn't be able to find a custom DTO
provided by a user.
In Camel we have found this problem several times and we've solved it by
changing the TCCL, or using the TCCL [1] [2] to resolve classes.
As I said, I need to look into this deeper and I'll have some time on
Thursday, so I hope you don't mind waiting a little bit. I have already
"bundle-fied" most of Ignite modules in the corresponding branch [3], so my
next step is to test out the Ignite examples inside an OSGi environment
(Karaf 4).
[1] https://issues.apache.org/jira/browse/CAMEL-5748
[2] https://issues.apache.org/jira/browse/CAMEL-5722
[3] https://github.com/apache/ignite/tree/ignite-1527
Regards,
*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk
On Tue, Nov 3, 2015 at 10:38 AM, Romain Gilles <ro...@gmail.com>
wrote:
> Hi Raul,
> - Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
> - I like the point 2. Maybe for a graph of object that come from different
> packages / bundles you may have to recursively capture the package version
> of the individual element of your graph.
> - For the point 3 I wonder if it will not over-complexify the solution. As
> a developer you will have to think about it. And it is not obvious in the
> code where things are serialized or not. You may use lambda in your
> application code therefore the current package become what you call a DTO
> package. The current state of ignite does not enforce you to specify it for
> "classical" classloading application. If you introduce this extra step for
> OSGi ready application you will maybe discourage people to use OSGi.
>
> To comeback to our solution. We start we a strong assumption: our cluster
> is homogeneous in term of code so, of course, it simplify our life :).
>
> BTW if you can introduce an extension point in the class resolution
> mechanism it can be interesting for end user in order to allow them to
> optimize it based on there specific use cases.
>
> Romain.
>
> Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <ra...@evosent.com> a écrit :
>
> > Hi Andrey,
> >
> > Thanks for the participation in this topic.
> >
> > I don't like the solution to incorporate the bundle symbolic name in the
> > serialised form. Nothing guarantees that the classdef will be located
> under
> > the same bundle in both source and target machines. We also have to take
> > into account that OSGi allows for multiple versions of the same
> > bundle/packages to co-exist in the same container. So it becomes more
> > complex.
> >
> > Using the TCCL should work when serialising, but it'll probably be of no
> > use when deserialising on the other end.
> >
> > I need to enhance Ignite to:
> >
> > 1. Use the TCCL when marshalling on one end.
> > 2. Incorporate the package version of the class in the serialised form if
> > Ignite is running in an OSGi environment.
> > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > through a custom OSGi manifest header or the like, as I explained before.
> > Package version must be taken into account.
> >
> > What do you think?
> >
> > Raúl.
> > On 2 Nov 2015 17:25, "Andrey Kornev" <an...@hotmail.com> wrote:
> >
> > > Raul,
> > >
> > > The fundamental hurdle we need to jump over to make Ignite OSGi-enabled
> > is
> > > the marshalling. More specifically the issue is with deserialization of
> > the
> > > classes that are provided by the bundles other than the Ignite bundle
> > > itself.
> > >
> > > When the Ignite transport layer receives a message it needs to figure
> out
> > > how to deserialize the bytes and for that it needs to know the bundle
> > that
> > > provides the class to be deserailized. At this point TCCL is of no use.
> > To
> > > make things more complex, the class may contain other classes that come
> > > from other bundles, and so on recursively. This means that each object
> in
> > > the hierarchy must be serialized with its bundle name (or bundle id),
> so
> > > that the deserializer will then be able to correctly resolve the class
> > > while traversing the object hierarchy during deserialization.
> > >
> > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> > plug
> > > in a custom class resolver. Romain's solution was to use Kryo that does
> > > provide a way to customize class resolution. It has solved the problem
> of
> > > capturing the bundle info and he was able to successfully run Ignite
> as a
> > > bundle in an OSGi container (after some repackaging and inclusion of
> the
> > > manifest). But Kryo-based marshalling introduced a lot of complexity to
> > the
> > > code and incorrect use of Kryo's numerous serializers caused some weird
> > > hard-to-debug issues in the Ignite core (like duplicate cache entries
> due
> > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > > figure!). Overall the Kryo-based solution is brittle and hard to
> > maintain.
> > >
> > > I feel the correct solution to OSGi problem would be to
> > > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > > original one) to be used in OSGi environment.
> > >
> > > Regards
> > > Andrey
> > >
> > > > From: raulk@apache.org
> > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > Subject: Re: OSGi migration may require a special marshaller
> > > > To: dev@ignite.apache.org
> > > >
> > > > Hi Romain,
> > > >
> > > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> > input.
> > > >
> > > > I'm thinking about the situation you describe and I wonder if you're
> > > > exporting Ignite as an OSGi service which is then consumed from other
> > > > bundles. Under this situation, it would be quite easy to reproduce
> the
> > > > behaviour you describe if Ignite is not resolving classes via the
> TCCL.
> > > > Need to dig deeper into that.
> > > >
> > > > Off the top of my head, there are two alternatives to solve it:
> > > >
> > > > 1. Use the TCCL for marshalling/unmarshalling (if not already used) –
> > we
> > > > gotta be wary of possible regressions.
> > > > 2. Create a special OSGi header Ignite-Export-Package so that bundles
> > > > containing DTOs can expose packages to Ignite's marshallers.
> > > >
> > > > Regards,
> > > >
> > > > *Raúl Kripalani*
> > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> > and
> > > > Messaging Engineer
> > > > http://about.me/raulkripalani |
> > http://www.linkedin.com/in/raulkripalani
> > > > http://blog.raulkr.net | twitter: @raulvk
> > > >
> > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > romain.gilles@misys.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'm really interested in this issue:
> > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff
> to
> > > make
> > > > > it work in our osgi environment. The main issue for us now is the
> > > > > serialization. I think it you will have to rework the
> > > OptimizedMarshaller
> > > > > or any other marshaller that works with object that come from
> outside
> > > your
> > > > > class space.
> > > > >
> > > > > We have try kryo that works. Kryo provide an extension point in
> order
> > > to
> > > > > resolve the classes:
> > > > >
> > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > . With this extension we are able to solve the problem of external
> > > classes.
> > > > > The only issue with kryo is that some classes need a certain care
> in
> > > the
> > > > > serialization process and therefore a specialized serializer.
> > > > >
> > > > > So I would like to know from the community what do think of
> changing
> > > the
> > > > > way the optimized marshaller works or introducing the support of
> yet
> > > > > another marshaller based on a kryo like technology?
> > > > >
> > > > >
> > > > > Thanks in advance,
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > >
> > > > > Romain.
> > > > >
> > > > >
> > > > > PS: I'm ready to help in the both cases.
> > > > > "Misys" is the trade name of the Misys group of companies. This
> email
> > > and
> > > > > any attachments have been scanned for known viruses using multiple
> > > > > scanners. This email message is intended for the named recipient
> > only.
> > > It
> > > > > may be privileged and/or confidential. If you are not the named
> > > recipient
> > > > > of this email please notify us immediately and do not copy it or
> use
> > > it for
> > > > > any purpose, nor disclose its contents to any other person. This
> > email
> > > does
> > > > > not constitute the commencement of legal relations between you and
> > > Misys.
> > > > > Please refer to the executed contract between you and the relevant
> > > member
> > > > > of the Misys group for the identity of the contracting party with
> > > which you
> > > > > are dealing.
> > > > >
> > >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Raul,
I think I see your point and agree. However, in Ignite 1.5 the
OptimizedMarshaller will be deprecated in favor of internal cross-platform
Binary format. I think we should wait till 1.5 is released and then
implement your fix in 1.6 within the scope of Ignite binary protocol.
Just one ask, let’s not use abbreviations in the method names (counter to
coding guidelines). How about setTransmitOsgiPackageVersion(…)?
D.
On Wed, Nov 11, 2015 at 5:31 AM, Raul Kripalani <ra...@apache.org> wrote:
> On Wed, Nov 11, 2015 at 1:48 AM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > Raul, we cannot be adding package name and version in all cases, as it
> will
> > have a negative impact on performance, especially when there is an
> > *optimistic* approach that requires no over-the-wire overhead whatsoever.
> >
>
> Why do we need to add the package name? The OptimizedObjectOutputStream
> already encodes the classname, which is then read on line 310 of
> OptimizedObjectInputStream (ignite-1.5 branch). Therefore, we already have
> the package name. All we need to encode is the version number which will be
> an additional String which can be further compressed:
>
> * For "1.0.0" we transmit "1".
> * For "1.1.0", we transmit "1.1".
> * etc.
>
> And this will only occur on a conditional basis when we detect we're
> running in an OSGi environment, although we need to cater for hybrid grid
> scenarios (comprising OSGi containers + non-OSGi containers). For all other
> cases, it would be skipped.
>
> If you'd still like to avoid these additional bytes, then a mode switch in
> OptimizedMarshaller: setTransmitOsgiPkgVer(boolean) should suffice.
>
> As I said, I don't see enough justification for pluggability in this
> context. I could see it if we were creating a mechanism for transmitting
> generic serialisation context. But everybody seems fixated on tackling
> classloader complexity only.
>
> I could also see some level of pluggability if this was applicable to any
> other marshallers. But how exactly do you see this being applicable to
> Kryo, Protobuf, or other potential ones (which we don't offer now)? How
> would you "encode the classloader" and where in the OutputStream? At the
> header? Before delegating the serialisation to the selected framework?
>
> Raul, personally I understand your sentiments, but to be honest I dislike
> > the names you are proposing even more. I still consider ClassLoaderCodec
> to
> > be the most elegant name here, considering all the other options I have
> > seen. It is concise and symmetric. I am open to changing it, but it must
> be
> > for the better, not for the worse.
>
>
> It's not about sentiment, it's about objectivity. The first step is to move
> away from the ClassLoaderCodec name, it's inaccurate. If anything I would
> propose SerializationContextCodec. But it implies extending its role to
> what I proposed: to provide and interpret information that assists the
> serialisation/deserialisation context.
>
> Let’s make sure that we follow the 80/20 rule and keep the default
> > implementation suitable for 80% of use cases with zero performance
> > overhead.
>
>
> No one questions that. The additional package version transmission would
> only kick in if we're in an OSGi environment.
>
> Having said that, I would be against artificially forcing this change into
> > 1.5, especially given the amount of controversy it generated.
>
>
> I don't think it's impossible to get it into 1.5 if that's the timeline.
>
> Regards,
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>
RE: OSGi migration may require a special marshaller
Posted by Andrey Kornev <an...@hotmail.com>.
Raul, if you don't mind, for posterity, could you please collect all your ideas spread all over this two-week long email thread and update the wiki page here https://cwiki.apache.org/confluence/display/IGNITE/OSGI+Compatibility? The details of your de-/serialization implementation would be especially interesting as personally I still don't understand what's being proposed. Also, please update the "Bundlezation" section with your approach to determining when an optional Ignite dependency should be a bundle and when a fragment.
Thanks a lot!
Andrey
> From: raulk@apache.org
> Date: Wed, 11 Nov 2015 13:31:04 +0000
> Subject: Re: OSGi migration may require a special marshaller
> To: dev@ignite.apache.org
>
> On Wed, Nov 11, 2015 at 1:48 AM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > Raul, we cannot be adding package name and version in all cases, as it will
> > have a negative impact on performance, especially when there is an
> > *optimistic* approach that requires no over-the-wire overhead whatsoever.
> >
>
> Why do we need to add the package name? The OptimizedObjectOutputStream
> already encodes the classname, which is then read on line 310 of
> OptimizedObjectInputStream (ignite-1.5 branch). Therefore, we already have
> the package name. All we need to encode is the version number which will be
> an additional String which can be further compressed:
>
> * For "1.0.0" we transmit "1".
> * For "1.1.0", we transmit "1.1".
> * etc.
>
> And this will only occur on a conditional basis when we detect we're
> running in an OSGi environment, although we need to cater for hybrid grid
> scenarios (comprising OSGi containers + non-OSGi containers). For all other
> cases, it would be skipped.
>
> If you'd still like to avoid these additional bytes, then a mode switch in
> OptimizedMarshaller: setTransmitOsgiPkgVer(boolean) should suffice.
>
> As I said, I don't see enough justification for pluggability in this
> context. I could see it if we were creating a mechanism for transmitting
> generic serialisation context. But everybody seems fixated on tackling
> classloader complexity only.
>
> I could also see some level of pluggability if this was applicable to any
> other marshallers. But how exactly do you see this being applicable to
> Kryo, Protobuf, or other potential ones (which we don't offer now)? How
> would you "encode the classloader" and where in the OutputStream? At the
> header? Before delegating the serialisation to the selected framework?
>
> Raul, personally I understand your sentiments, but to be honest I dislike
> > the names you are proposing even more. I still consider ClassLoaderCodec to
> > be the most elegant name here, considering all the other options I have
> > seen. It is concise and symmetric. I am open to changing it, but it must be
> > for the better, not for the worse.
>
>
> It's not about sentiment, it's about objectivity. The first step is to move
> away from the ClassLoaderCodec name, it's inaccurate. If anything I would
> propose SerializationContextCodec. But it implies extending its role to
> what I proposed: to provide and interpret information that assists the
> serialisation/deserialisation context.
>
> Let’s make sure that we follow the 80/20 rule and keep the default
> > implementation suitable for 80% of use cases with zero performance
> > overhead.
>
>
> No one questions that. The additional package version transmission would
> only kick in if we're in an OSGi environment.
>
> Having said that, I would be against artificially forcing this change into
> > 1.5, especially given the amount of controversy it generated.
>
>
> I don't think it's impossible to get it into 1.5 if that's the timeline.
>
> Regards,
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
Re: OSGi migration may require a special marshaller
Posted by Raul Kripalani <ra...@apache.org>.
On Wed, Nov 11, 2015 at 1:48 AM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> Raul, we cannot be adding package name and version in all cases, as it will
> have a negative impact on performance, especially when there is an
> *optimistic* approach that requires no over-the-wire overhead whatsoever.
>
Why do we need to add the package name? The OptimizedObjectOutputStream
already encodes the classname, which is then read on line 310 of
OptimizedObjectInputStream (ignite-1.5 branch). Therefore, we already have
the package name. All we need to encode is the version number which will be
an additional String which can be further compressed:
* For "1.0.0" we transmit "1".
* For "1.1.0", we transmit "1.1".
* etc.
And this will only occur on a conditional basis when we detect we're
running in an OSGi environment, although we need to cater for hybrid grid
scenarios (comprising OSGi containers + non-OSGi containers). For all other
cases, it would be skipped.
If you'd still like to avoid these additional bytes, then a mode switch in
OptimizedMarshaller: setTransmitOsgiPkgVer(boolean) should suffice.
As I said, I don't see enough justification for pluggability in this
context. I could see it if we were creating a mechanism for transmitting
generic serialisation context. But everybody seems fixated on tackling
classloader complexity only.
I could also see some level of pluggability if this was applicable to any
other marshallers. But how exactly do you see this being applicable to
Kryo, Protobuf, or other potential ones (which we don't offer now)? How
would you "encode the classloader" and where in the OutputStream? At the
header? Before delegating the serialisation to the selected framework?
Raul, personally I understand your sentiments, but to be honest I dislike
> the names you are proposing even more. I still consider ClassLoaderCodec to
> be the most elegant name here, considering all the other options I have
> seen. It is concise and symmetric. I am open to changing it, but it must be
> for the better, not for the worse.
It's not about sentiment, it's about objectivity. The first step is to move
away from the ClassLoaderCodec name, it's inaccurate. If anything I would
propose SerializationContextCodec. But it implies extending its role to
what I proposed: to provide and interpret information that assists the
serialisation/deserialisation context.
Let’s make sure that we follow the 80/20 rule and keep the default
> implementation suitable for 80% of use cases with zero performance
> overhead.
No one questions that. The additional package version transmission would
only kick in if we're in an OSGi environment.
Having said that, I would be against artificially forcing this change into
> 1.5, especially given the amount of controversy it generated.
I don't think it's impossible to get it into 1.5 if that's the timeline.
Regards,
*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk
Re: OSGi migration may require a special marshaller
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Gentlemen,
I have several comments below…
On Tue, Nov 10, 2015 at 11:52 AM, Raul Kripalani <ra...@apache.org> wrote:
> Andrey, I already specified some points of my disagreement in the Wiki. In
> my previous email I also said: "If we create an SPI it should be for more
> generic hinting applicable in other circumstances during serialisation with
> other frameworks and what not."
>
We should not use the SPI acronym liberally, as SPIs have a totally
different meaning in Ignite. The ClassLoaderCodec should just be a setter
on IgniteConfiguration, not a full-blown SPI.
> What I don't understand is the interest in an SPI for something that
> appears to have been solved by just transmitting the package version and
> using the correct logic on the receiving side to locate the appropriate
> Bundle and its classloader, as I explained in the wiki. That's a fully
> deterministic way of solving this problem.
Raul, we cannot be adding package name and version in all cases, as it will
have a negative impact on performance, especially when there is an
*optimistic* approach that requires no over-the-wire overhead whatsoever.
As I already mentioned before, the default implementation *must* be the
optimistic one, since Ignite is measured based on its performance first and
foremost.
This is the main reason why we actually need an API like ClassLoaderCodec,
so it can have more than one implementation.
> To me, creating an SPI targeted to OSGi class loading doesn't solve any
> problem. We don’t need that much flexibility. A different story would be to
> provide a mechanism to compute generic serialisation/deserialisation
> context data, but you guys already rejected this need in the Wiki.
>
I don’t see how a ClassLoaderCodec is specific to OSGI at all. It does not
have OSGI in its name and should be useful in other cases where custom
class-loading is required.
I would say, however, that ClassLoaderCodec should be specific to
class-loader detection only, and we should not overburden it with other
semantics solved elsewhere in Ignite. For example, custom
serialization/deserialization is handled in Binaryzable interface, much in
the same way as Java solves it with Externalizable interface.
Raul, if you disagree, can you please provide your reasoning here?
> With regards to the naming, sorry to be blunt, but ClassLoaderCodec is an
> awful name. It implies that we are serialising a class loader. Wikipedia
> definition for Codec: "A codec is a device or computer program capable of
> encoding or decoding a digital data stream or signal." ClassLoaderCodec
> would mean "a coder/decoder for a ClassLoader". And we would have not been
> doing that; we would have just calculated and transmitted context data to
> aid deserialisation, i.e. "hints", not a classloader.
>
Naming discussions are always passionate :)
Raul, personally I understand your sentiments, but to be honest I dislike
the names you are proposing even more. I still consider ClassLoaderCodec to
be the most elegant name here, considering all the other options I have
seen. It is concise and symmetric. I am open to changing it, but it must be
for the better, not for the worse.
> If you don't mind, could we please pause this discussion until I get a
> proof of concept working these days? And then we can discuss over an actual
> implementation to get consensus. I'd rather invest my time in that manner.
>
I see your point, but I still felt compelled to reply, especially because
of possible performance impact of the proposed design. Let’s make sure that
we follow the 80/20 rule and keep the default implementation suitable for
80% of use cases with zero performance overhead.
> When is Ignite 1.5 due?
>
Given what I am seeing in the 1.5 release thread, I think we are
realistically looking at the end of next week. Having said that, I would be
against artificially forcing this change into 1.5, especially given the
amount of controversy it generated.
Let’s postpone it for 1.6.
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>
> On Tue, Nov 10, 2015 at 7:23 PM, Andrey Kornev <an...@hotmail.com>
> wrote:
>
> > Raul,
> >
> > Could you please be a bit more specific about the nature of your
> > disagreement? Is the proposed SPI not generic enough? Or, is it just the
> > naming?
> >
> > As per Romain's suggestion, could you please just make sure the SPI is
> > hooked into the new marshalling implementation and a no-op codec
> > implementation is available in time for the 1.5 release? This would be
> the
> > first step toward full OSGi support in 1.6.
> >
> > Thanks
> > Andrey
> >
> > > From: raulk@apache.org
> > > Date: Tue, 10 Nov 2015 18:22:59 +0000
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: dev@ignite.apache.org
> > >
> > > Hi Romain,
> > >
> > > The implementation I have in mind won't be costly. I'm working on it
> this
> > > week.
> > >
> > > I still don't agree with a ClassLoaderCodec SPI as-is. If we create an
> > SPI
> > > it should be for more generic hinting applicable in other circumstances
> > > during serialisation with other frameworks and what not.
> > >
> > > Otherwise, we should not create a pluggable SPI at all and simply
> > implement
> > > an option. As a matter of fact, with the last solution I proposed I
> don't
> > > think there will be any edge cases at all where users would need to
> > create
> > > their own "ClassLoaderCodec" at all.
> > >
> > > Regards,
> > >
> > > *Raúl Kripalani*
> > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> and
> > > Messaging Engineer
> > > http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> > > http://blog.raulkr.net | twitter: @raulvk
> > >
> > > On Tue, Nov 10, 2015 at 3:06 PM, Romain Gilles <
> romain.gilles@gmail.com>
> > > wrote:
> > >
> > > > To be honest Raoul, I'm more interesting in the interface (SPI)
> > declaration
> > > > than the implementation. If you can do it for the 1.5 I'm ok.
> > Otherwise I
> > > > ready to start it by introducing the interface and implementing the
> > dummy
> > > > version for non OSGi environment.
> > > >
> > > > It will free me to implement a temporary OSGi version by the time you
> > are
> > > > done with the overall IGNITE-1270 tasks.
> > > >
> > > > Thanks,
> > > >
> > > > Romain
> > > >
> > > > Le mar. 10 nov. 2015 à 15:42, Raul Kripalani <ra...@evosent.com> a
> > écrit :
> > > >
> > > > > Hi Romain,
> > > > >
> > > > > If you don't mind, I'm working on the entire OSGi integration
> > including
> > > > the
> > > > > serialisation technique.
> > > > >
> > > > > I'll ping you if I need help.
> > > > >
> > > > > Thanks for your collaboration,
> > > > >
> > > > > *Raúl Kripalani*
> > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> > Data and
> > > > > Messaging Engineer
> > > > > http://about.me/raulkripalani |
> > http://www.linkedin.com/in/raulkripalani
> > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > >
> > > > > On Tue, Nov 10, 2015 at 1:50 PM, Romain Gilles <
> > romain.gilles@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > I have put my comments. Hope they will make the things progress
> :)
> > > > > > Should I start to implement this ticket or should I wait and see
> if
> > > > Raoul
> > > > > > want to take it?
> > > > > >
> > > > > > Romain
> > > > > >
> > > > > > Le mar. 10 nov. 2015 à 02:58, Dmitriy Setrakyan <
> > dsetrakyan@apache.org
> > > > >
> > > > > a
> > > > > > écrit :
> > > > > >
> > > > > > > Thanks Raul, great points! I have created a ticket for the
> > > > > > > class-loader-detection design, described on wiki:
> > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-1877
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Mon, Nov 9, 2015 at 3:42 PM, Raul Kripalani <
> raulk@apache.org
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hey Romain,
> > > > > > > >
> > > > > > > > I've updated the design proposal in the Wiki with my input.
> > Could
> > > > you
> > > > > > > > please have a look and let me know what you think?
> > > > > > > >
> > > > > > > > I'll implement a proof of concept of my proposed
> > > > > > > marshalling/unmarshalling
> > > > > > > > strategy and push it to Git so you can take it for a spin.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > *Raúl Kripalani*
> > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration,
> > Big
> > > > Data
> > > > > > and
> > > > > > > > Messaging Engineer
> > > > > > > > http://about.me/raulkripalani |
> > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > > >
> > > > > > > > On Fri, Nov 6, 2015 at 4:05 PM, Romain Gilles <
> > > > > romain.gilles@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > > I will put some sample code this WE. I'm exhausted sorry
> for
> > the
> > > > > > > delay...
> > > > > > > > > Romain
> > > > > > > > >
> > > > > > > > > Le ven. 6 nov. 2015 à 00:45, Andrey Kornev <
> > > > > andrewkornev@hotmail.com
> > > > > > >
> > > > > > > a
> > > > > > > > > écrit :
> > > > > > > > >
> > > > > > > > > > Hello,
> > > > > > > > > >
> > > > > > > > > > We've made an attempt to formalize the approach to
> Ignite's
> > > > OSGi
> > > > > > > > > > enablement:
> > > > > > > > > >
> > > > > > >
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/OSGI+Compatibility.
> > > > > > > > > > Please have a look and feel free to provide your positive
> > > > > feedback
> > > > > > > :)))
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Andrey
> > > > > > > > > >
> > > > > > > > > > > From: dsetrakyan@apache.org
> > > > > > > > > > > Date: Wed, 4 Nov 2015 09:26:17 -0800
> > > > > > > > > > > Subject: Re: OSGi migration may require a special
> > marshaller
> > > > > > > > > > > To: dev@ignite.apache.org
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Nov 4, 2015 at 9:07 AM, Romain Gilles <
> > > > > > > > romain.gilles@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Dmitriy,
> > > > > > > > > > > > I found this post that explain how to find a bundle
> > based
> > > > on
> > > > > > its
> > > > > > > > > bundle
> > > > > > > > > > > > name and version.
> > > > > > > > > > > > This post explain for the past to now how to do it in
> > the
> > > > > > > standard
> > > > > > > > > way
> > > > > > > > > > with
> > > > > > > > > > > > "pull" approach:
> > > > > > > > https://www.eclipse.org/forums/index.php/t/205719/
> > > > > > > > > > > > Regarding the version of OSGi you want to support
> then
> > some
> > > > > > > > solutions
> > > > > > > > > > will
> > > > > > > > > > > > works some others will not.
> > > > > > > > > > > > There is an other way to do this stuff without using
> > those
> > > > > > "pull"
> > > > > > > > > style
> > > > > > > > > > > > approach based on BundleTracker. If you want I can
> > give you
> > > > > the
> > > > > > > > code
> > > > > > > > > > to do
> > > > > > > > > > > > it with BundlTracker. I think with this solution you
> > will
> > > > > > > support a
> > > > > > > > > > wider
> > > > > > > > > > > > range of OSGi version.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Romain, if you can provide a generic code sample to
> look
> > up a
> > > > > > > > > ClassLoader
> > > > > > > > > > > in OSGI based on manifest properties, would be great.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Raul Kripalani <ra...@apache.org>.
Andrey, I already specified some points of my disagreement in the Wiki. In
my previous email I also said: "If we create an SPI it should be for more
generic hinting applicable in other circumstances during serialisation with
other frameworks and what not."
What I don't understand is the interest in an SPI for something that
appears to have been solved by just transmitting the package version and
using the correct logic on the receiving side to locate the appropriate
Bundle and its classloader, as I explained in the wiki. That's a fully
deterministic way of solving this problem. To me, creating an SPI targeted
to OSGi classloading doesn't solve any problem. We don't need that much
flexibility. A different story would be to provide a mechanism to compute
generic serialisation/deserialisation context data, but you guys already
rejected this need in the Wiki.
With regards to the naming, sorry to be blunt, but ClassLoaderCodec is an
awful name. It implies that we are serialising a class loader. Wikipedia
definition for Codec: "A codec is a device or computer program capable of
encoding or decoding a digital data stream or signal." ClassLoaderCodec
would mean "a coder/decoder for a ClassLoader". And we would have not been
doing that; we would have just calculated and transmitted context data to
aid deserialisation, i.e. "hints", not a classloader.
If you don't mind, could we please pause this discussion until I get a
proof of concept working these days? And then we can discuss over an actual
implementation to get consensus. I'd rather invest my time in that manner.
When is Ignite 1.5 due?
*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk
On Tue, Nov 10, 2015 at 7:23 PM, Andrey Kornev <an...@hotmail.com>
wrote:
> Raul,
>
> Could you please be a bit more specific about the nature of your
> disagreement? Is the proposed SPI not generic enough? Or, is it just the
> naming?
>
> As per Romain's suggestion, could you please just make sure the SPI is
> hooked into the new marshalling implementation and a no-op codec
> implementation is available in time for the 1.5 release? This would be the
> first step toward full OSGi support in 1.6.
>
> Thanks
> Andrey
>
> > From: raulk@apache.org
> > Date: Tue, 10 Nov 2015 18:22:59 +0000
> > Subject: Re: OSGi migration may require a special marshaller
> > To: dev@ignite.apache.org
> >
> > Hi Romain,
> >
> > The implementation I have in mind won't be costly. I'm working on it this
> > week.
> >
> > I still don't agree with a ClassLoaderCodec SPI as-is. If we create an
> SPI
> > it should be for more generic hinting applicable in other circumstances
> > during serialisation with other frameworks and what not.
> >
> > Otherwise, we should not create a pluggable SPI at all and simply
> implement
> > an option. As a matter of fact, with the last solution I proposed I don't
> > think there will be any edge cases at all where users would need to
> create
> > their own "ClassLoaderCodec" at all.
> >
> > Regards,
> >
> > *Raúl Kripalani*
> > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> > Messaging Engineer
> > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > http://blog.raulkr.net | twitter: @raulvk
> >
> > On Tue, Nov 10, 2015 at 3:06 PM, Romain Gilles <ro...@gmail.com>
> > wrote:
> >
> > > To be honest Raoul, I'm more interesting in the interface (SPI)
> declaration
> > > than the implementation. If you can do it for the 1.5 I'm ok.
> Otherwise I
> > > ready to start it by introducing the interface and implementing the
> dummy
> > > version for non OSGi environment.
> > >
> > > It will free me to implement a temporary OSGi version by the time you
> are
> > > done with the overall IGNITE-1270 tasks.
> > >
> > > Thanks,
> > >
> > > Romain
> > >
> > > Le mar. 10 nov. 2015 à 15:42, Raul Kripalani <ra...@evosent.com> a
> écrit :
> > >
> > > > Hi Romain,
> > > >
> > > > If you don't mind, I'm working on the entire OSGi integration
> including
> > > the
> > > > serialisation technique.
> > > >
> > > > I'll ping you if I need help.
> > > >
> > > > Thanks for your collaboration,
> > > >
> > > > *Raúl Kripalani*
> > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> Data and
> > > > Messaging Engineer
> > > > http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> > > > http://blog.raulkr.net | twitter: @raulvk
> > > >
> > > > On Tue, Nov 10, 2015 at 1:50 PM, Romain Gilles <
> romain.gilles@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > I have put my comments. Hope they will make the things progress :)
> > > > > Should I start to implement this ticket or should I wait and see if
> > > Raoul
> > > > > want to take it?
> > > > >
> > > > > Romain
> > > > >
> > > > > Le mar. 10 nov. 2015 à 02:58, Dmitriy Setrakyan <
> dsetrakyan@apache.org
> > > >
> > > > a
> > > > > écrit :
> > > > >
> > > > > > Thanks Raul, great points! I have created a ticket for the
> > > > > > class-loader-detection design, described on wiki:
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/IGNITE-1877
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Mon, Nov 9, 2015 at 3:42 PM, Raul Kripalani <raulk@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > > > Hey Romain,
> > > > > > >
> > > > > > > I've updated the design proposal in the Wiki with my input.
> Could
> > > you
> > > > > > > please have a look and let me know what you think?
> > > > > > >
> > > > > > > I'll implement a proof of concept of my proposed
> > > > > > marshalling/unmarshalling
> > > > > > > strategy and push it to Git so you can take it for a spin.
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > *Raúl Kripalani*
> > > > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration,
> Big
> > > Data
> > > > > and
> > > > > > > Messaging Engineer
> > > > > > > http://about.me/raulkripalani |
> > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > >
> > > > > > > On Fri, Nov 6, 2015 at 4:05 PM, Romain Gilles <
> > > > romain.gilles@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > > I will put some sample code this WE. I'm exhausted sorry for
> the
> > > > > > delay...
> > > > > > > > Romain
> > > > > > > >
> > > > > > > > Le ven. 6 nov. 2015 à 00:45, Andrey Kornev <
> > > > andrewkornev@hotmail.com
> > > > > >
> > > > > > a
> > > > > > > > écrit :
> > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > >
> > > > > > > > > We've made an attempt to formalize the approach to Ignite's
> > > OSGi
> > > > > > > > > enablement:
> > > > > > > > >
> > > > > >
> > > https://cwiki.apache.org/confluence/display/IGNITE/OSGI+Compatibility.
> > > > > > > > > Please have a look and feel free to provide your positive
> > > > feedback
> > > > > > :)))
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Andrey
> > > > > > > > >
> > > > > > > > > > From: dsetrakyan@apache.org
> > > > > > > > > > Date: Wed, 4 Nov 2015 09:26:17 -0800
> > > > > > > > > > Subject: Re: OSGi migration may require a special
> marshaller
> > > > > > > > > > To: dev@ignite.apache.org
> > > > > > > > > >
> > > > > > > > > > On Wed, Nov 4, 2015 at 9:07 AM, Romain Gilles <
> > > > > > > romain.gilles@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Dmitriy,
> > > > > > > > > > > I found this post that explain how to find a bundle
> based
> > > on
> > > > > its
> > > > > > > > bundle
> > > > > > > > > > > name and version.
> > > > > > > > > > > This post explain for the past to now how to do it in
> the
> > > > > > standard
> > > > > > > > way
> > > > > > > > > with
> > > > > > > > > > > "pull" approach:
> > > > > > > https://www.eclipse.org/forums/index.php/t/205719/
> > > > > > > > > > > Regarding the version of OSGi you want to support then
> some
> > > > > > > solutions
> > > > > > > > > will
> > > > > > > > > > > works some others will not.
> > > > > > > > > > > There is an other way to do this stuff without using
> those
> > > > > "pull"
> > > > > > > > style
> > > > > > > > > > > approach based on BundleTracker. If you want I can
> give you
> > > > the
> > > > > > > code
> > > > > > > > > to do
> > > > > > > > > > > it with BundlTracker. I think with this solution you
> will
> > > > > > support a
> > > > > > > > > wider
> > > > > > > > > > > range of OSGi version.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Romain, if you can provide a generic code sample to look
> up a
> > > > > > > > ClassLoader
> > > > > > > > > > in OSGI based on manifest properties, would be great.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
>
RE: OSGi migration may require a special marshaller
Posted by Andrey Kornev <an...@hotmail.com>.
Raul,
Could you please be a bit more specific about the nature of your disagreement? Is the proposed SPI not generic enough? Or, is it just the naming?
As per Romain's suggestion, could you please just make sure the SPI is hooked into the new marshalling implementation and a no-op codec implementation is available in time for the 1.5 release? This would be the first step toward full OSGi support in 1.6.
Thanks
Andrey
> From: raulk@apache.org
> Date: Tue, 10 Nov 2015 18:22:59 +0000
> Subject: Re: OSGi migration may require a special marshaller
> To: dev@ignite.apache.org
>
> Hi Romain,
>
> The implementation I have in mind won't be costly. I'm working on it this
> week.
>
> I still don't agree with a ClassLoaderCodec SPI as-is. If we create an SPI
> it should be for more generic hinting applicable in other circumstances
> during serialisation with other frameworks and what not.
>
> Otherwise, we should not create a pluggable SPI at all and simply implement
> an option. As a matter of fact, with the last solution I proposed I don't
> think there will be any edge cases at all where users would need to create
> their own "ClassLoaderCodec" at all.
>
> Regards,
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>
> On Tue, Nov 10, 2015 at 3:06 PM, Romain Gilles <ro...@gmail.com>
> wrote:
>
> > To be honest Raoul, I'm more interesting in the interface (SPI) declaration
> > than the implementation. If you can do it for the 1.5 I'm ok. Otherwise I
> > ready to start it by introducing the interface and implementing the dummy
> > version for non OSGi environment.
> >
> > It will free me to implement a temporary OSGi version by the time you are
> > done with the overall IGNITE-1270 tasks.
> >
> > Thanks,
> >
> > Romain
> >
> > Le mar. 10 nov. 2015 à 15:42, Raul Kripalani <ra...@evosent.com> a écrit :
> >
> > > Hi Romain,
> > >
> > > If you don't mind, I'm working on the entire OSGi integration including
> > the
> > > serialisation technique.
> > >
> > > I'll ping you if I need help.
> > >
> > > Thanks for your collaboration,
> > >
> > > *Raúl Kripalani*
> > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> > > Messaging Engineer
> > > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > > http://blog.raulkr.net | twitter: @raulvk
> > >
> > > On Tue, Nov 10, 2015 at 1:50 PM, Romain Gilles <ro...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > > I have put my comments. Hope they will make the things progress :)
> > > > Should I start to implement this ticket or should I wait and see if
> > Raoul
> > > > want to take it?
> > > >
> > > > Romain
> > > >
> > > > Le mar. 10 nov. 2015 à 02:58, Dmitriy Setrakyan <dsetrakyan@apache.org
> > >
> > > a
> > > > écrit :
> > > >
> > > > > Thanks Raul, great points! I have created a ticket for the
> > > > > class-loader-detection design, described on wiki:
> > > > >
> > > > > https://issues.apache.org/jira/browse/IGNITE-1877
> > > > >
> > > > > D.
> > > > >
> > > > > On Mon, Nov 9, 2015 at 3:42 PM, Raul Kripalani <ra...@apache.org>
> > > wrote:
> > > > >
> > > > > > Hey Romain,
> > > > > >
> > > > > > I've updated the design proposal in the Wiki with my input. Could
> > you
> > > > > > please have a look and let me know what you think?
> > > > > >
> > > > > > I'll implement a proof of concept of my proposed
> > > > > marshalling/unmarshalling
> > > > > > strategy and push it to Git so you can take it for a spin.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > *Raúl Kripalani*
> > > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> > Data
> > > > and
> > > > > > Messaging Engineer
> > > > > > http://about.me/raulkripalani |
> > > > http://www.linkedin.com/in/raulkripalani
> > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > >
> > > > > > On Fri, Nov 6, 2015 at 4:05 PM, Romain Gilles <
> > > romain.gilles@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > I will put some sample code this WE. I'm exhausted sorry for the
> > > > > delay...
> > > > > > > Romain
> > > > > > >
> > > > > > > Le ven. 6 nov. 2015 à 00:45, Andrey Kornev <
> > > andrewkornev@hotmail.com
> > > > >
> > > > > a
> > > > > > > écrit :
> > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > We've made an attempt to formalize the approach to Ignite's
> > OSGi
> > > > > > > > enablement:
> > > > > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/OSGI+Compatibility.
> > > > > > > > Please have a look and feel free to provide your positive
> > > feedback
> > > > > :)))
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Andrey
> > > > > > > >
> > > > > > > > > From: dsetrakyan@apache.org
> > > > > > > > > Date: Wed, 4 Nov 2015 09:26:17 -0800
> > > > > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > > > > To: dev@ignite.apache.org
> > > > > > > > >
> > > > > > > > > On Wed, Nov 4, 2015 at 9:07 AM, Romain Gilles <
> > > > > > romain.gilles@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Dmitriy,
> > > > > > > > > > I found this post that explain how to find a bundle based
> > on
> > > > its
> > > > > > > bundle
> > > > > > > > > > name and version.
> > > > > > > > > > This post explain for the past to now how to do it in the
> > > > > standard
> > > > > > > way
> > > > > > > > with
> > > > > > > > > > "pull" approach:
> > > > > > https://www.eclipse.org/forums/index.php/t/205719/
> > > > > > > > > > Regarding the version of OSGi you want to support then some
> > > > > > solutions
> > > > > > > > will
> > > > > > > > > > works some others will not.
> > > > > > > > > > There is an other way to do this stuff without using those
> > > > "pull"
> > > > > > > style
> > > > > > > > > > approach based on BundleTracker. If you want I can give you
> > > the
> > > > > > code
> > > > > > > > to do
> > > > > > > > > > it with BundlTracker. I think with this solution you will
> > > > > support a
> > > > > > > > wider
> > > > > > > > > > range of OSGi version.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Romain, if you can provide a generic code sample to look up a
> > > > > > > ClassLoader
> > > > > > > > > in OSGI based on manifest properties, would be great.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
Re: OSGi migration may require a special marshaller
Posted by Raul Kripalani <ra...@apache.org>.
Hi Romain,
The implementation I have in mind won't be costly. I'm working on it this
week.
I still don't agree with a ClassLoaderCodec SPI as-is. If we create an SPI
it should be for more generic hinting applicable in other circumstances
during serialisation with other frameworks and what not.
Otherwise, we should not create a pluggable SPI at all and simply implement
an option. As a matter of fact, with the last solution I proposed I don't
think there will be any edge cases at all where users would need to create
their own "ClassLoaderCodec" at all.
Regards,
*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk
On Tue, Nov 10, 2015 at 3:06 PM, Romain Gilles <ro...@gmail.com>
wrote:
> To be honest Raoul, I'm more interesting in the interface (SPI) declaration
> than the implementation. If you can do it for the 1.5 I'm ok. Otherwise I
> ready to start it by introducing the interface and implementing the dummy
> version for non OSGi environment.
>
> It will free me to implement a temporary OSGi version by the time you are
> done with the overall IGNITE-1270 tasks.
>
> Thanks,
>
> Romain
>
> Le mar. 10 nov. 2015 à 15:42, Raul Kripalani <ra...@evosent.com> a écrit :
>
> > Hi Romain,
> >
> > If you don't mind, I'm working on the entire OSGi integration including
> the
> > serialisation technique.
> >
> > I'll ping you if I need help.
> >
> > Thanks for your collaboration,
> >
> > *Raúl Kripalani*
> > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> > Messaging Engineer
> > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > http://blog.raulkr.net | twitter: @raulvk
> >
> > On Tue, Nov 10, 2015 at 1:50 PM, Romain Gilles <ro...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I have put my comments. Hope they will make the things progress :)
> > > Should I start to implement this ticket or should I wait and see if
> Raoul
> > > want to take it?
> > >
> > > Romain
> > >
> > > Le mar. 10 nov. 2015 à 02:58, Dmitriy Setrakyan <dsetrakyan@apache.org
> >
> > a
> > > écrit :
> > >
> > > > Thanks Raul, great points! I have created a ticket for the
> > > > class-loader-detection design, described on wiki:
> > > >
> > > > https://issues.apache.org/jira/browse/IGNITE-1877
> > > >
> > > > D.
> > > >
> > > > On Mon, Nov 9, 2015 at 3:42 PM, Raul Kripalani <ra...@apache.org>
> > wrote:
> > > >
> > > > > Hey Romain,
> > > > >
> > > > > I've updated the design proposal in the Wiki with my input. Could
> you
> > > > > please have a look and let me know what you think?
> > > > >
> > > > > I'll implement a proof of concept of my proposed
> > > > marshalling/unmarshalling
> > > > > strategy and push it to Git so you can take it for a spin.
> > > > >
> > > > > Regards,
> > > > >
> > > > > *Raúl Kripalani*
> > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> Data
> > > and
> > > > > Messaging Engineer
> > > > > http://about.me/raulkripalani |
> > > http://www.linkedin.com/in/raulkripalani
> > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > >
> > > > > On Fri, Nov 6, 2015 at 4:05 PM, Romain Gilles <
> > romain.gilles@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > I will put some sample code this WE. I'm exhausted sorry for the
> > > > delay...
> > > > > > Romain
> > > > > >
> > > > > > Le ven. 6 nov. 2015 à 00:45, Andrey Kornev <
> > andrewkornev@hotmail.com
> > > >
> > > > a
> > > > > > écrit :
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > We've made an attempt to formalize the approach to Ignite's
> OSGi
> > > > > > > enablement:
> > > > > > >
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/OSGI+Compatibility.
> > > > > > > Please have a look and feel free to provide your positive
> > feedback
> > > > :)))
> > > > > > >
> > > > > > > Thanks
> > > > > > > Andrey
> > > > > > >
> > > > > > > > From: dsetrakyan@apache.org
> > > > > > > > Date: Wed, 4 Nov 2015 09:26:17 -0800
> > > > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > > > To: dev@ignite.apache.org
> > > > > > > >
> > > > > > > > On Wed, Nov 4, 2015 at 9:07 AM, Romain Gilles <
> > > > > romain.gilles@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Dmitriy,
> > > > > > > > > I found this post that explain how to find a bundle based
> on
> > > its
> > > > > > bundle
> > > > > > > > > name and version.
> > > > > > > > > This post explain for the past to now how to do it in the
> > > > standard
> > > > > > way
> > > > > > > with
> > > > > > > > > "pull" approach:
> > > > > https://www.eclipse.org/forums/index.php/t/205719/
> > > > > > > > > Regarding the version of OSGi you want to support then some
> > > > > solutions
> > > > > > > will
> > > > > > > > > works some others will not.
> > > > > > > > > There is an other way to do this stuff without using those
> > > "pull"
> > > > > > style
> > > > > > > > > approach based on BundleTracker. If you want I can give you
> > the
> > > > > code
> > > > > > > to do
> > > > > > > > > it with BundlTracker. I think with this solution you will
> > > > support a
> > > > > > > wider
> > > > > > > > > range of OSGi version.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Romain, if you can provide a generic code sample to look up a
> > > > > > ClassLoader
> > > > > > > > in OSGI based on manifest properties, would be great.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Romain Gilles <ro...@gmail.com>.
To be honest Raoul, I'm more interesting in the interface (SPI) declaration
than the implementation. If you can do it for the 1.5 I'm ok. Otherwise I
ready to start it by introducing the interface and implementing the dummy
version for non OSGi environment.
It will free me to implement a temporary OSGi version by the time you are
done with the overall IGNITE-1270 tasks.
Thanks,
Romain
Le mar. 10 nov. 2015 à 15:42, Raul Kripalani <ra...@evosent.com> a écrit :
> Hi Romain,
>
> If you don't mind, I'm working on the entire OSGi integration including the
> serialisation technique.
>
> I'll ping you if I need help.
>
> Thanks for your collaboration,
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>
> On Tue, Nov 10, 2015 at 1:50 PM, Romain Gilles <ro...@gmail.com>
> wrote:
>
> > Hi,
> > I have put my comments. Hope they will make the things progress :)
> > Should I start to implement this ticket or should I wait and see if Raoul
> > want to take it?
> >
> > Romain
> >
> > Le mar. 10 nov. 2015 à 02:58, Dmitriy Setrakyan <ds...@apache.org>
> a
> > écrit :
> >
> > > Thanks Raul, great points! I have created a ticket for the
> > > class-loader-detection design, described on wiki:
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-1877
> > >
> > > D.
> > >
> > > On Mon, Nov 9, 2015 at 3:42 PM, Raul Kripalani <ra...@apache.org>
> wrote:
> > >
> > > > Hey Romain,
> > > >
> > > > I've updated the design proposal in the Wiki with my input. Could you
> > > > please have a look and let me know what you think?
> > > >
> > > > I'll implement a proof of concept of my proposed
> > > marshalling/unmarshalling
> > > > strategy and push it to Git so you can take it for a spin.
> > > >
> > > > Regards,
> > > >
> > > > *Raúl Kripalani*
> > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> > and
> > > > Messaging Engineer
> > > > http://about.me/raulkripalani |
> > http://www.linkedin.com/in/raulkripalani
> > > > http://blog.raulkr.net | twitter: @raulvk
> > > >
> > > > On Fri, Nov 6, 2015 at 4:05 PM, Romain Gilles <
> romain.gilles@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > I will put some sample code this WE. I'm exhausted sorry for the
> > > delay...
> > > > > Romain
> > > > >
> > > > > Le ven. 6 nov. 2015 à 00:45, Andrey Kornev <
> andrewkornev@hotmail.com
> > >
> > > a
> > > > > écrit :
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > We've made an attempt to formalize the approach to Ignite's OSGi
> > > > > > enablement:
> > > > > >
> > > https://cwiki.apache.org/confluence/display/IGNITE/OSGI+Compatibility.
> > > > > > Please have a look and feel free to provide your positive
> feedback
> > > :)))
> > > > > >
> > > > > > Thanks
> > > > > > Andrey
> > > > > >
> > > > > > > From: dsetrakyan@apache.org
> > > > > > > Date: Wed, 4 Nov 2015 09:26:17 -0800
> > > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > > To: dev@ignite.apache.org
> > > > > > >
> > > > > > > On Wed, Nov 4, 2015 at 9:07 AM, Romain Gilles <
> > > > romain.gilles@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Dmitriy,
> > > > > > > > I found this post that explain how to find a bundle based on
> > its
> > > > > bundle
> > > > > > > > name and version.
> > > > > > > > This post explain for the past to now how to do it in the
> > > standard
> > > > > way
> > > > > > with
> > > > > > > > "pull" approach:
> > > > https://www.eclipse.org/forums/index.php/t/205719/
> > > > > > > > Regarding the version of OSGi you want to support then some
> > > > solutions
> > > > > > will
> > > > > > > > works some others will not.
> > > > > > > > There is an other way to do this stuff without using those
> > "pull"
> > > > > style
> > > > > > > > approach based on BundleTracker. If you want I can give you
> the
> > > > code
> > > > > > to do
> > > > > > > > it with BundlTracker. I think with this solution you will
> > > support a
> > > > > > wider
> > > > > > > > range of OSGi version.
> > > > > > > >
> > > > > > >
> > > > > > > Romain, if you can provide a generic code sample to look up a
> > > > > ClassLoader
> > > > > > > in OSGI based on manifest properties, would be great.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Raul Kripalani <ra...@evosent.com>.
Hi Romain,
If you don't mind, I'm working on the entire OSGi integration including the
serialisation technique.
I'll ping you if I need help.
Thanks for your collaboration,
*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk
On Tue, Nov 10, 2015 at 1:50 PM, Romain Gilles <ro...@gmail.com>
wrote:
> Hi,
> I have put my comments. Hope they will make the things progress :)
> Should I start to implement this ticket or should I wait and see if Raoul
> want to take it?
>
> Romain
>
> Le mar. 10 nov. 2015 à 02:58, Dmitriy Setrakyan <ds...@apache.org> a
> écrit :
>
> > Thanks Raul, great points! I have created a ticket for the
> > class-loader-detection design, described on wiki:
> >
> > https://issues.apache.org/jira/browse/IGNITE-1877
> >
> > D.
> >
> > On Mon, Nov 9, 2015 at 3:42 PM, Raul Kripalani <ra...@apache.org> wrote:
> >
> > > Hey Romain,
> > >
> > > I've updated the design proposal in the Wiki with my input. Could you
> > > please have a look and let me know what you think?
> > >
> > > I'll implement a proof of concept of my proposed
> > marshalling/unmarshalling
> > > strategy and push it to Git so you can take it for a spin.
> > >
> > > Regards,
> > >
> > > *Raúl Kripalani*
> > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> and
> > > Messaging Engineer
> > > http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> > > http://blog.raulkr.net | twitter: @raulvk
> > >
> > > On Fri, Nov 6, 2015 at 4:05 PM, Romain Gilles <romain.gilles@gmail.com
> >
> > > wrote:
> > >
> > > > Hi,
> > > > I will put some sample code this WE. I'm exhausted sorry for the
> > delay...
> > > > Romain
> > > >
> > > > Le ven. 6 nov. 2015 à 00:45, Andrey Kornev <andrewkornev@hotmail.com
> >
> > a
> > > > écrit :
> > > >
> > > > > Hello,
> > > > >
> > > > > We've made an attempt to formalize the approach to Ignite's OSGi
> > > > > enablement:
> > > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/OSGI+Compatibility.
> > > > > Please have a look and feel free to provide your positive feedback
> > :)))
> > > > >
> > > > > Thanks
> > > > > Andrey
> > > > >
> > > > > > From: dsetrakyan@apache.org
> > > > > > Date: Wed, 4 Nov 2015 09:26:17 -0800
> > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > To: dev@ignite.apache.org
> > > > > >
> > > > > > On Wed, Nov 4, 2015 at 9:07 AM, Romain Gilles <
> > > romain.gilles@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Dmitriy,
> > > > > > > I found this post that explain how to find a bundle based on
> its
> > > > bundle
> > > > > > > name and version.
> > > > > > > This post explain for the past to now how to do it in the
> > standard
> > > > way
> > > > > with
> > > > > > > "pull" approach:
> > > https://www.eclipse.org/forums/index.php/t/205719/
> > > > > > > Regarding the version of OSGi you want to support then some
> > > solutions
> > > > > will
> > > > > > > works some others will not.
> > > > > > > There is an other way to do this stuff without using those
> "pull"
> > > > style
> > > > > > > approach based on BundleTracker. If you want I can give you the
> > > code
> > > > > to do
> > > > > > > it with BundlTracker. I think with this solution you will
> > support a
> > > > > wider
> > > > > > > range of OSGi version.
> > > > > > >
> > > > > >
> > > > > > Romain, if you can provide a generic code sample to look up a
> > > > ClassLoader
> > > > > > in OSGI based on manifest properties, would be great.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Romain Gilles <ro...@gmail.com>.
Hi,
I have put my comments. Hope they will make the things progress :)
Should I start to implement this ticket or should I wait and see if Raoul
want to take it?
Romain
Le mar. 10 nov. 2015 à 02:58, Dmitriy Setrakyan <ds...@apache.org> a
écrit :
> Thanks Raul, great points! I have created a ticket for the
> class-loader-detection design, described on wiki:
>
> https://issues.apache.org/jira/browse/IGNITE-1877
>
> D.
>
> On Mon, Nov 9, 2015 at 3:42 PM, Raul Kripalani <ra...@apache.org> wrote:
>
> > Hey Romain,
> >
> > I've updated the design proposal in the Wiki with my input. Could you
> > please have a look and let me know what you think?
> >
> > I'll implement a proof of concept of my proposed
> marshalling/unmarshalling
> > strategy and push it to Git so you can take it for a spin.
> >
> > Regards,
> >
> > *Raúl Kripalani*
> > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> > Messaging Engineer
> > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > http://blog.raulkr.net | twitter: @raulvk
> >
> > On Fri, Nov 6, 2015 at 4:05 PM, Romain Gilles <ro...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I will put some sample code this WE. I'm exhausted sorry for the
> delay...
> > > Romain
> > >
> > > Le ven. 6 nov. 2015 à 00:45, Andrey Kornev <an...@hotmail.com>
> a
> > > écrit :
> > >
> > > > Hello,
> > > >
> > > > We've made an attempt to formalize the approach to Ignite's OSGi
> > > > enablement:
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/OSGI+Compatibility.
> > > > Please have a look and feel free to provide your positive feedback
> :)))
> > > >
> > > > Thanks
> > > > Andrey
> > > >
> > > > > From: dsetrakyan@apache.org
> > > > > Date: Wed, 4 Nov 2015 09:26:17 -0800
> > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > To: dev@ignite.apache.org
> > > > >
> > > > > On Wed, Nov 4, 2015 at 9:07 AM, Romain Gilles <
> > romain.gilles@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Dmitriy,
> > > > > > I found this post that explain how to find a bundle based on its
> > > bundle
> > > > > > name and version.
> > > > > > This post explain for the past to now how to do it in the
> standard
> > > way
> > > > with
> > > > > > "pull" approach:
> > https://www.eclipse.org/forums/index.php/t/205719/
> > > > > > Regarding the version of OSGi you want to support then some
> > solutions
> > > > will
> > > > > > works some others will not.
> > > > > > There is an other way to do this stuff without using those "pull"
> > > style
> > > > > > approach based on BundleTracker. If you want I can give you the
> > code
> > > > to do
> > > > > > it with BundlTracker. I think with this solution you will
> support a
> > > > wider
> > > > > > range of OSGi version.
> > > > > >
> > > > >
> > > > > Romain, if you can provide a generic code sample to look up a
> > > ClassLoader
> > > > > in OSGI based on manifest properties, would be great.
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Thanks Raul, great points! I have created a ticket for the
class-loader-detection design, described on wiki:
https://issues.apache.org/jira/browse/IGNITE-1877
D.
On Mon, Nov 9, 2015 at 3:42 PM, Raul Kripalani <ra...@apache.org> wrote:
> Hey Romain,
>
> I've updated the design proposal in the Wiki with my input. Could you
> please have a look and let me know what you think?
>
> I'll implement a proof of concept of my proposed marshalling/unmarshalling
> strategy and push it to Git so you can take it for a spin.
>
> Regards,
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>
> On Fri, Nov 6, 2015 at 4:05 PM, Romain Gilles <ro...@gmail.com>
> wrote:
>
> > Hi,
> > I will put some sample code this WE. I'm exhausted sorry for the delay...
> > Romain
> >
> > Le ven. 6 nov. 2015 à 00:45, Andrey Kornev <an...@hotmail.com> a
> > écrit :
> >
> > > Hello,
> > >
> > > We've made an attempt to formalize the approach to Ignite's OSGi
> > > enablement:
> > > https://cwiki.apache.org/confluence/display/IGNITE/OSGI+Compatibility.
> > > Please have a look and feel free to provide your positive feedback :)))
> > >
> > > Thanks
> > > Andrey
> > >
> > > > From: dsetrakyan@apache.org
> > > > Date: Wed, 4 Nov 2015 09:26:17 -0800
> > > > Subject: Re: OSGi migration may require a special marshaller
> > > > To: dev@ignite.apache.org
> > > >
> > > > On Wed, Nov 4, 2015 at 9:07 AM, Romain Gilles <
> romain.gilles@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Dmitriy,
> > > > > I found this post that explain how to find a bundle based on its
> > bundle
> > > > > name and version.
> > > > > This post explain for the past to now how to do it in the standard
> > way
> > > with
> > > > > "pull" approach:
> https://www.eclipse.org/forums/index.php/t/205719/
> > > > > Regarding the version of OSGi you want to support then some
> solutions
> > > will
> > > > > works some others will not.
> > > > > There is an other way to do this stuff without using those "pull"
> > style
> > > > > approach based on BundleTracker. If you want I can give you the
> code
> > > to do
> > > > > it with BundlTracker. I think with this solution you will support a
> > > wider
> > > > > range of OSGi version.
> > > > >
> > > >
> > > > Romain, if you can provide a generic code sample to look up a
> > ClassLoader
> > > > in OSGI based on manifest properties, would be great.
> > > >
> > > >
> > >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Raul Kripalani <ra...@apache.org>.
Hey Romain,
I've updated the design proposal in the Wiki with my input. Could you
please have a look and let me know what you think?
I'll implement a proof of concept of my proposed marshalling/unmarshalling
strategy and push it to Git so you can take it for a spin.
Regards,
*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk
On Fri, Nov 6, 2015 at 4:05 PM, Romain Gilles <ro...@gmail.com>
wrote:
> Hi,
> I will put some sample code this WE. I'm exhausted sorry for the delay...
> Romain
>
> Le ven. 6 nov. 2015 à 00:45, Andrey Kornev <an...@hotmail.com> a
> écrit :
>
> > Hello,
> >
> > We've made an attempt to formalize the approach to Ignite's OSGi
> > enablement:
> > https://cwiki.apache.org/confluence/display/IGNITE/OSGI+Compatibility.
> > Please have a look and feel free to provide your positive feedback :)))
> >
> > Thanks
> > Andrey
> >
> > > From: dsetrakyan@apache.org
> > > Date: Wed, 4 Nov 2015 09:26:17 -0800
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: dev@ignite.apache.org
> > >
> > > On Wed, Nov 4, 2015 at 9:07 AM, Romain Gilles <romain.gilles@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Dmitriy,
> > > > I found this post that explain how to find a bundle based on its
> bundle
> > > > name and version.
> > > > This post explain for the past to now how to do it in the standard
> way
> > with
> > > > "pull" approach: https://www.eclipse.org/forums/index.php/t/205719/
> > > > Regarding the version of OSGi you want to support then some solutions
> > will
> > > > works some others will not.
> > > > There is an other way to do this stuff without using those "pull"
> style
> > > > approach based on BundleTracker. If you want I can give you the code
> > to do
> > > > it with BundlTracker. I think with this solution you will support a
> > wider
> > > > range of OSGi version.
> > > >
> > >
> > > Romain, if you can provide a generic code sample to look up a
> ClassLoader
> > > in OSGI based on manifest properties, would be great.
> > >
> > >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Romain Gilles <ro...@gmail.com>.
Hi,
I will put some sample code this WE. I'm exhausted sorry for the delay...
Romain
Le ven. 6 nov. 2015 à 00:45, Andrey Kornev <an...@hotmail.com> a
écrit :
> Hello,
>
> We've made an attempt to formalize the approach to Ignite's OSGi
> enablement:
> https://cwiki.apache.org/confluence/display/IGNITE/OSGI+Compatibility.
> Please have a look and feel free to provide your positive feedback :)))
>
> Thanks
> Andrey
>
> > From: dsetrakyan@apache.org
> > Date: Wed, 4 Nov 2015 09:26:17 -0800
> > Subject: Re: OSGi migration may require a special marshaller
> > To: dev@ignite.apache.org
> >
> > On Wed, Nov 4, 2015 at 9:07 AM, Romain Gilles <ro...@gmail.com>
> > wrote:
> >
> > > Hi Dmitriy,
> > > I found this post that explain how to find a bundle based on its bundle
> > > name and version.
> > > This post explain for the past to now how to do it in the standard way
> with
> > > "pull" approach: https://www.eclipse.org/forums/index.php/t/205719/
> > > Regarding the version of OSGi you want to support then some solutions
> will
> > > works some others will not.
> > > There is an other way to do this stuff without using those "pull" style
> > > approach based on BundleTracker. If you want I can give you the code
> to do
> > > it with BundlTracker. I think with this solution you will support a
> wider
> > > range of OSGi version.
> > >
> >
> > Romain, if you can provide a generic code sample to look up a ClassLoader
> > in OSGI based on manifest properties, would be great.
> >
> >
>
RE: OSGi migration may require a special marshaller
Posted by Andrey Kornev <an...@hotmail.com>.
Hello,
We've made an attempt to formalize the approach to Ignite's OSGi enablement: https://cwiki.apache.org/confluence/display/IGNITE/OSGI+Compatibility. Please have a look and feel free to provide your positive feedback :)))
Thanks
Andrey
> From: dsetrakyan@apache.org
> Date: Wed, 4 Nov 2015 09:26:17 -0800
> Subject: Re: OSGi migration may require a special marshaller
> To: dev@ignite.apache.org
>
> On Wed, Nov 4, 2015 at 9:07 AM, Romain Gilles <ro...@gmail.com>
> wrote:
>
> > Hi Dmitriy,
> > I found this post that explain how to find a bundle based on its bundle
> > name and version.
> > This post explain for the past to now how to do it in the standard way with
> > "pull" approach: https://www.eclipse.org/forums/index.php/t/205719/
> > Regarding the version of OSGi you want to support then some solutions will
> > works some others will not.
> > There is an other way to do this stuff without using those "pull" style
> > approach based on BundleTracker. If you want I can give you the code to do
> > it with BundlTracker. I think with this solution you will support a wider
> > range of OSGi version.
> >
>
> Romain, if you can provide a generic code sample to look up a ClassLoader
> in OSGI based on manifest properties, would be great.
>
>
Re: OSGi migration may require a special marshaller
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Wed, Nov 4, 2015 at 9:07 AM, Romain Gilles <ro...@gmail.com>
wrote:
> Hi Dmitriy,
> I found this post that explain how to find a bundle based on its bundle
> name and version.
> This post explain for the past to now how to do it in the standard way with
> "pull" approach: https://www.eclipse.org/forums/index.php/t/205719/
> Regarding the version of OSGi you want to support then some solutions will
> works some others will not.
> There is an other way to do this stuff without using those "pull" style
> approach based on BundleTracker. If you want I can give you the code to do
> it with BundlTracker. I think with this solution you will support a wider
> range of OSGi version.
>
Romain, if you can provide a generic code sample to look up a ClassLoader
in OSGI based on manifest properties, would be great.
>
> Le mer. 4 nov. 2015 à 04:07, Dmitriy Setrakyan <ds...@apache.org> a
> écrit :
>
> > Andrey, et al,
> >
> > Can you provide a way on how to get a required Bundle object based on,
> say,
> > bundle symbolic name and version?
> >
> > I have reached the end of the internet trying to find a way in OSGI to
> look
> > up a Bundle based on something other than an actual Class belonging to
> that
> > bundle, but no luck.
> >
> > D.
> >
> > On Tue, Nov 3, 2015 at 5:15 PM, Andrey Kornev <an...@hotmail.com>
> > wrote:
> >
> > > Dmitriy,
> > >
> > > I think your approach will work, but I let Romain respond.
> > >
> > > Also, in terms of the implementation, please keep in mind that the
> > > resolver must be called for each non-JDK and non-Ignite core class (it
> > > would probably make sense to eschew storing class loaders for such
> > classes
> > > in favor of compactness of the serialized representation -- see below).
> > > Also, it's worth keeping a cache of already resolved class loaders per
> > > marshaller invocation (this is probably the context that Romain has
> > > mentioned in his previous posting) to minimize the number of the
> resolver
> > > calls.
> > >
> > > In terms of the resolver's implementation, the simplest way to
> serialize
> > > the class loader would be by capturing two pieces of information (both
> > > strings): the bundle symbolic name and the bundle version. This
> approach
> > > however may result in bloating of the serialized representation: I'd
> > > roughly estimate the overhead per element to be at least 20-30 bytes
> (the
> > > length of the symbolic name string, plus the length of the version
> > string).
> > > There are way to reduce the overhead (like serializing the hash code of
> > the
> > > bundle id string rather than the string itself, and then come up with a
> > > clever way of resolving the hash collisions), but they all come at cost
> > of
> > > increased complexity...
> > >
> > > An alternative approach would be rely on the special bundle id which is
> > an
> > > integer and is generated by the OSGi container. But in this case, all
> > nodes
> > > must ensure that all the bundles have consistent ids (bundle A with id
> 42
> > > on node N1, has the same id on every other node) which is not easy --
> > while
> > > not entirely impossible -- to guarantee. As long as the nodes are
> > > homogeneous (have the same set of bundles deployed) the OSGi container
> is
> > > guaranteed to assign to the bundles the same ids.
> > >
> > > Thanks
> > > Andrey
> > >
> > > > From: dsetrakyan@apache.org
> > > > Date: Tue, 3 Nov 2015 16:29:41 -0800
> > > > Subject: Re: OSGi migration may require a special marshaller
> > > > To: dev@ignite.apache.org
> > > >
> > > > Romain,
> > > >
> > > > In the upcoming release we will be deprecating the
> OptimizedMarshaller
> > > and
> > > > will be switching to a default internal marshaller (which is based on
> > the
> > > > new PortableMarshaller donated by GridGain).
> > > >
> > > > Having said that, we may be able to pass BinaryWriter and
> BinaryReader
> > > > instead of byte arrays. This will be pretty close to passing the
> > stream,
> > > as
> > > > suggested by Andrey.
> > > >
> > > > Also, I still think that we should only resolve class loaders and not
> > the
> > > > class itself. The main reason is that Ignite will encode class names
> > into
> > > > an integer hash code and will store the integer->class-fqn mapping in
> > > > internal replicated cache. I doubt users will get more efficient than
> > an
> > > > integer (4 bytes) for a class name.
> > > >
> > > > On the receiving side, once we are able to get the right class
> loader,
> > we
> > > > can easily get the proper class by calling
> > > ClassLoader.findClass(class-fqn).
> > > >
> > > > Thoughts?
> > > >
> > > > D.
> > > >
> > > > On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <
> romain.gilles@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > Maybe a missing point. I think but I'm not sure that in the
> > > > > OptimizedMarshaller there is a caching of already serialized
> classes.
> > > Due
> > > > > to the dynamic nature of OSGi this may lead to memory leak. In fact
> > if
> > > a
> > > > > bundle is refreshed, it will produce a new BundleRevision and
> > > therefore a
> > > > > new classloader. And if you don't release the class from the
> previous
> > > > > BundleRevision then you endup with memory leak. So maybe the
> > Marshaller
> > > > > interface or somewhere should provide a way to free those classes /
> > > > > classloaders.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Romain
> > > > >
> > > > > Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <
> andrewkornev@hotmail.com
> > >
> > > a
> > > > > écrit :
> > > > >
> > > > > > Romain/Dmitriy,
> > > > > >
> > > > > > I prefer Romain's approach, but just curious, in the API you guys
> > are
> > > > > > proposing why use a byte[] rather than OutputStream/InputStream?
> > > With a
> > > > > > byte[], one would inevitably end up wrapping it into a byte
> stream
> > > class.
> > > > > > Also, the stream-based interface would be more in line with the
> > > > > Marshaller
> > > > > > API.
> > > > > >
> > > > > > Also for symmetry with the readClass() method, I suggest the
> > > writeClass()
> > > > > > take a Class<?> rather than an object.
> > > > > >
> > > > > > Regards
> > > > > > Andrey
> > > > > >
> > > > > > > From: romain.gilles@gmail.com
> > > > > > > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > > To: dev@ignite.apache.org
> > > > > > >
> > > > > > > Hi Dmitriy,
> > > > > > > I think your solution is good. Maybe I will change it a little
> > > bit...
> > > > > :P
> > > > > > > I think you should delegate the Class resolution to the
> resolver.
> > > > > Because
> > > > > > > for example with your current solution the marshaller may
> before
> > > (or
> > > > > > after)
> > > > > > > store the fqn of the class (maybe only the first time it
> > encounter
> > > it)
> > > > > > but
> > > > > > > in order to save the classloader context resolution the
> > > implementation
> > > > > of
> > > > > > > the resolver may have to save the package name of the given
> > object
> > > and
> > > > > > some
> > > > > > > extra info therefore the content package name will be
> serialized
> > > two
> > > > > > times.
> > > > > > > Ok, it's not a big deal.
> > > > > > > But now if the resolver identify that it can reuse the same
> class
> > > > > loader
> > > > > > > for a couple of classes. It will be interesting for it to have
> a
> > > > > > > serialization context in order to save, let say, classloader/id
> > > mapping
> > > > > > in
> > > > > > > order to save the id instead of a more longer representation.
> > > > > > > So I propose something like that:
> > > > > > > *public interface ClassResolver {*
> > > > > > > * // This method is invoked on the sender side to *
> > > > > > > * // marshal the information about the class.*
> > > > > > > * // where the context is a map style object that is
> reset/new
> > > for
> > > > > > each
> > > > > > > object graph serialization.*
> > > > > > > * public byte[] writeClass(Object o, Context context) throws
> > > > > > > IgniteException;*
> > > > > > >
> > > > > > > * // This method is invoked on the receiving side to*
> > > > > > > * // retrieve the class based on provided information.*
> > > > > > > * // where the context is a map style object that is
> reset/new
> > > for
> > > > > > each
> > > > > > > object graph serialization.*
> > > > > > > * public Class<?> readClass(byte[], Context context) throws
> > > > > > > IgniteException;*
> > > > > > > *}*
> > > > > > >
> > > > > > > I think your proposal will solve our issue and maybe also open
> a
> > > door
> > > > > for
> > > > > > > the osgi development.
> > > > > > > Let me know what do you think about me proposal? :)
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Romain
> > > > > > >
> > > > > > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org>
> > > > > a
> > > > > > > écrit :
> > > > > > >
> > > > > > > > Romain,
> > > > > > > >
> > > > > > > > Can you comment on the ClassLoaderResolver suggestion that I
> > > proposed
> > > > > > > > earlier? Will it solve your problem?
> > > > > > > >
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <
> > > > > romain.gilles@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Raul,
> > > > > > > > > - Do you plan to use the TCCL when marshalling in OSGi
> env?
> > > If yes
> > > > > > how?
> > > > > > > > > - I like the point 2. Maybe for a graph of object that
> come
> > > from
> > > > > > > > different
> > > > > > > > > packages / bundles you may have to recursively capture the
> > > package
> > > > > > > > version
> > > > > > > > > of the individual element of your graph.
> > > > > > > > > - For the point 3 I wonder if it will not over-complexify
> > the
> > > > > > solution.
> > > > > > > > As
> > > > > > > > > a developer you will have to think about it. And it is not
> > > obvious
> > > > > > in the
> > > > > > > > > code where things are serialized or not. You may use lambda
> > in
> > > your
> > > > > > > > > application code therefore the current package become what
> > you
> > > call
> > > > > > a DTO
> > > > > > > > > package. The current state of ignite does not enforce you
> to
> > > > > specify
> > > > > > it
> > > > > > > > for
> > > > > > > > > "classical" classloading application. If you introduce this
> > > extra
> > > > > > step
> > > > > > > > for
> > > > > > > > > OSGi ready application you will maybe discourage people to
> > use
> > > > > OSGi.
> > > > > > > > >
> > > > > > > > > To comeback to our solution. We start we a strong
> assumption:
> > > our
> > > > > > cluster
> > > > > > > > > is homogeneous in term of code so, of course, it simplify
> our
> > > life
> > > > > > :).
> > > > > > > > >
> > > > > > > > > BTW if you can introduce an extension point in the class
> > > resolution
> > > > > > > > > mechanism it can be interesting for end user in order to
> > allow
> > > them
> > > > > > to
> > > > > > > > > optimize it based on there specific use cases.
> > > > > > > > >
> > > > > > > > > Romain.
> > > > > > > > >
> > > > > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <
> > raul@evosent.com>
> > > a
> > > > > > écrit :
> > > > > > > > >
> > > > > > > > > > Hi Andrey,
> > > > > > > > > >
> > > > > > > > > > Thanks for the participation in this topic.
> > > > > > > > > >
> > > > > > > > > > I don't like the solution to incorporate the bundle
> > symbolic
> > > name
> > > > > > in
> > > > > > > > the
> > > > > > > > > > serialised form. Nothing guarantees that the classdef
> will
> > be
> > > > > > located
> > > > > > > > > under
> > > > > > > > > > the same bundle in both source and target machines. We
> also
> > > have
> > > > > to
> > > > > > > > take
> > > > > > > > > > into account that OSGi allows for multiple versions of
> the
> > > same
> > > > > > > > > > bundle/packages to co-exist in the same container. So it
> > > becomes
> > > > > > more
> > > > > > > > > > complex.
> > > > > > > > > >
> > > > > > > > > > Using the TCCL should work when serialising, but it'll
> > > probably
> > > > > be
> > > > > > of
> > > > > > > > no
> > > > > > > > > > use when deserialising on the other end.
> > > > > > > > > >
> > > > > > > > > > I need to enhance Ignite to:
> > > > > > > > > >
> > > > > > > > > > 1. Use the TCCL when marshalling on one end.
> > > > > > > > > > 2. Incorporate the package version of the class in the
> > > serialised
> > > > > > form
> > > > > > > > if
> > > > > > > > > > Ignite is running in an OSGi environment.
> > > > > > > > > > 3. On the receiver end, discover cache entities / DTOs in
> > all
> > > > > > bundles
> > > > > > > > > > through a custom OSGi manifest header or the like, as I
> > > explained
> > > > > > > > before.
> > > > > > > > > > Package version must be taken into account.
> > > > > > > > > >
> > > > > > > > > > What do you think?
> > > > > > > > > >
> > > > > > > > > > Raúl.
> > > > > > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <
> > > andrewkornev@hotmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Raul,
> > > > > > > > > > >
> > > > > > > > > > > The fundamental hurdle we need to jump over to make
> > Ignite
> > > > > > > > OSGi-enabled
> > > > > > > > > > is
> > > > > > > > > > > the marshalling. More specifically the issue is with
> > > > > > deserialization
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > classes that are provided by the bundles other than the
> > > Ignite
> > > > > > bundle
> > > > > > > > > > > itself.
> > > > > > > > > > >
> > > > > > > > > > > When the Ignite transport layer receives a message it
> > > needs to
> > > > > > figure
> > > > > > > > > out
> > > > > > > > > > > how to deserialize the bytes and for that it needs to
> > know
> > > the
> > > > > > bundle
> > > > > > > > > > that
> > > > > > > > > > > provides the class to be deserailized. At this point
> TCCL
> > > is of
> > > > > > no
> > > > > > > > use.
> > > > > > > > > > To
> > > > > > > > > > > make things more complex, the class may contain other
> > > classes
> > > > > > that
> > > > > > > > come
> > > > > > > > > > > from other bundles, and so on recursively. This means
> > that
> > > each
> > > > > > > > object
> > > > > > > > > in
> > > > > > > > > > > the hierarchy must be serialized with its bundle name
> (or
> > > > > bundle
> > > > > > id),
> > > > > > > > > so
> > > > > > > > > > > that the deserializer will then be able to correctly
> > > resolve
> > > > > the
> > > > > > > > class
> > > > > > > > > > > while traversing the object hierarchy during
> > > deserialization.
> > > > > > > > > > >
> > > > > > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking
> > the
> > > > > > ability to
> > > > > > > > > > plug
> > > > > > > > > > > in a custom class resolver. Romain's solution was to
> use
> > > Kryo
> > > > > > that
> > > > > > > > does
> > > > > > > > > > > provide a way to customize class resolution. It has
> > solved
> > > the
> > > > > > > > problem
> > > > > > > > > of
> > > > > > > > > > > capturing the bundle info and he was able to
> successfully
> > > run
> > > > > > Ignite
> > > > > > > > > as a
> > > > > > > > > > > bundle in an OSGi container (after some repackaging and
> > > > > > inclusion of
> > > > > > > > > the
> > > > > > > > > > > manifest). But Kryo-based marshalling introduced a lot
> of
> > > > > > complexity
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > code and incorrect use of Kryo's numerous serializers
> > > caused
> > > > > some
> > > > > > > > weird
> > > > > > > > > > > hard-to-debug issues in the Ignite core (like duplicate
> > > cache
> > > > > > entries
> > > > > > > > > due
> > > > > > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap
> > > class --
> > > > > > go
> > > > > > > > > > > figure!). Overall the Kryo-based solution is brittle
> and
> > > hard
> > > > > to
> > > > > > > > > > maintain.
> > > > > > > > > > >
> > > > > > > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > > > > > > 1) enhance the OptimizedMarshaller to allow custom
> class
> > > > > > resolution.
> > > > > > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in
> > > addition to
> > > > > > the
> > > > > > > > > > > original one) to be used in OSGi environment.
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > > Andrey
> > > > > > > > > > >
> > > > > > > > > > > > From: raulk@apache.org
> > > > > > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > > > > > > Subject: Re: OSGi migration may require a special
> > > marshaller
> > > > > > > > > > > > To: dev@ignite.apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Romain,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm working on the OSGi compatibility of Ignite. I
> > > appreciate
> > > > > > your
> > > > > > > > > > input.
> > > > > > > > > > > >
> > > > > > > > > > > > I'm thinking about the situation you describe and I
> > > wonder if
> > > > > > > > you're
> > > > > > > > > > > > exporting Ignite as an OSGi service which is then
> > > consumed
> > > > > from
> > > > > > > > other
> > > > > > > > > > > > bundles. Under this situation, it would be quite easy
> > to
> > > > > > reproduce
> > > > > > > > > the
> > > > > > > > > > > > behaviour you describe if Ignite is not resolving
> > > classes via
> > > > > > the
> > > > > > > > > TCCL.
> > > > > > > > > > > > Need to dig deeper into that.
> > > > > > > > > > > >
> > > > > > > > > > > > Off the top of my head, there are two alternatives to
> > > solve
> > > > > it:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not
> > > already
> > > > > > > > used) –
> > > > > > > > > > we
> > > > > > > > > > > > gotta be wary of possible regressions.
> > > > > > > > > > > > 2. Create a special OSGi header Ignite-Export-Package
> > so
> > > that
> > > > > > > > bundles
> > > > > > > > > > > > containing DTOs can expose packages to Ignite's
> > > marshallers.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > >
> > > > > > > > > > > > *Raúl Kripalani*
> > > > > > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel |
> > > Integration,
> > > > > > Big
> > > > > > > > Data
> > > > > > > > > > and
> > > > > > > > > > > > Messaging Engineer
> > > > > > > > > > > > http://about.me/raulkripalani |
> > > > > > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > > > > > > romain.gilles@misys.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm really interested in this issue:
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270
> .
> > We
> > > > > some
> > > > > > > > stuff
> > > > > > > > > to
> > > > > > > > > > > make
> > > > > > > > > > > > > it work in our osgi environment. The main issue for
> > us
> > > now
> > > > > > is the
> > > > > > > > > > > > > serialization. I think it you will have to rework
> the
> > > > > > > > > > > OptimizedMarshaller
> > > > > > > > > > > > > or any other marshaller that works with object that
> > > come
> > > > > from
> > > > > > > > > outside
> > > > > > > > > > > your
> > > > > > > > > > > > > class space.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We have try kryo that works. Kryo provide an
> > extension
> > > > > point
> > > > > > in
> > > > > > > > > order
> > > > > > > > > > > to
> > > > > > > > > > > > > resolve the classes:
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > > > > > > . With this extension we are able to solve the
> > problem
> > > of
> > > > > > > > external
> > > > > > > > > > > classes.
> > > > > > > > > > > > > The only issue with kryo is that some classes need
> a
> > > > > certain
> > > > > > care
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > serialization process and therefore a specialized
> > > > > serializer.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So I would like to know from the community what do
> > > think of
> > > > > > > > > changing
> > > > > > > > > > > the
> > > > > > > > > > > > > way the optimized marshaller works or introducing
> the
> > > > > > support of
> > > > > > > > > yet
> > > > > > > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks in advance,
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Romain.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > > > > > > "Misys" is the trade name of the Misys group of
> > > companies.
> > > > > > This
> > > > > > > > > email
> > > > > > > > > > > and
> > > > > > > > > > > > > any attachments have been scanned for known viruses
> > > using
> > > > > > > > multiple
> > > > > > > > > > > > > scanners. This email message is intended for the
> > named
> > > > > > recipient
> > > > > > > > > > only.
> > > > > > > > > > > It
> > > > > > > > > > > > > may be privileged and/or confidential. If you are
> not
> > > the
> > > > > > named
> > > > > > > > > > > recipient
> > > > > > > > > > > > > of this email please notify us immediately and do
> not
> > > copy
> > > > > > it or
> > > > > > > > > use
> > > > > > > > > > > it for
> > > > > > > > > > > > > any purpose, nor disclose its contents to any other
> > > person.
> > > > > > This
> > > > > > > > > > email
> > > > > > > > > > > does
> > > > > > > > > > > > > not constitute the commencement of legal relations
> > > between
> > > > > > you
> > > > > > > > and
> > > > > > > > > > > Misys.
> > > > > > > > > > > > > Please refer to the executed contract between you
> and
> > > the
> > > > > > > > relevant
> > > > > > > > > > > member
> > > > > > > > > > > > > of the Misys group for the identity of the
> > contracting
> > > > > party
> > > > > > with
> > > > > > > > > > > which you
> > > > > > > > > > > > > are dealing.
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> > >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Romain Gilles <ro...@gmail.com>.
Hi Dmitriy,
I found this post that explain how to find a bundle based on its bundle
name and version.
This post explain for the past to now how to do it in the standard way with
"pull" approach: https://www.eclipse.org/forums/index.php/t/205719/
Regarding the version of OSGi you want to support then some solutions will
works some others will not.
There is an other way to do this stuff without using those "pull" style
approach based on BundleTracker. If you want I can give you the code to do
it with BundlTracker. I think with this solution you will support a wider
range of OSGi version.
Romain
Le mer. 4 nov. 2015 à 04:07, Dmitriy Setrakyan <ds...@apache.org> a
écrit :
> Andrey, et al,
>
> Can you provide a way on how to get a required Bundle object based on, say,
> bundle symbolic name and version?
>
> I have reached the end of the internet trying to find a way in OSGI to look
> up a Bundle based on something other than an actual Class belonging to that
> bundle, but no luck.
>
> D.
>
> On Tue, Nov 3, 2015 at 5:15 PM, Andrey Kornev <an...@hotmail.com>
> wrote:
>
> > Dmitriy,
> >
> > I think your approach will work, but I let Romain respond.
> >
> > Also, in terms of the implementation, please keep in mind that the
> > resolver must be called for each non-JDK and non-Ignite core class (it
> > would probably make sense to eschew storing class loaders for such
> classes
> > in favor of compactness of the serialized representation -- see below).
> > Also, it's worth keeping a cache of already resolved class loaders per
> > marshaller invocation (this is probably the context that Romain has
> > mentioned in his previous posting) to minimize the number of the resolver
> > calls.
> >
> > In terms of the resolver's implementation, the simplest way to serialize
> > the class loader would be by capturing two pieces of information (both
> > strings): the bundle symbolic name and the bundle version. This approach
> > however may result in bloating of the serialized representation: I'd
> > roughly estimate the overhead per element to be at least 20-30 bytes (the
> > length of the symbolic name string, plus the length of the version
> string).
> > There are way to reduce the overhead (like serializing the hash code of
> the
> > bundle id string rather than the string itself, and then come up with a
> > clever way of resolving the hash collisions), but they all come at cost
> of
> > increased complexity...
> >
> > An alternative approach would be rely on the special bundle id which is
> an
> > integer and is generated by the OSGi container. But in this case, all
> nodes
> > must ensure that all the bundles have consistent ids (bundle A with id 42
> > on node N1, has the same id on every other node) which is not easy --
> while
> > not entirely impossible -- to guarantee. As long as the nodes are
> > homogeneous (have the same set of bundles deployed) the OSGi container is
> > guaranteed to assign to the bundles the same ids.
> >
> > Thanks
> > Andrey
> >
> > > From: dsetrakyan@apache.org
> > > Date: Tue, 3 Nov 2015 16:29:41 -0800
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: dev@ignite.apache.org
> > >
> > > Romain,
> > >
> > > In the upcoming release we will be deprecating the OptimizedMarshaller
> > and
> > > will be switching to a default internal marshaller (which is based on
> the
> > > new PortableMarshaller donated by GridGain).
> > >
> > > Having said that, we may be able to pass BinaryWriter and BinaryReader
> > > instead of byte arrays. This will be pretty close to passing the
> stream,
> > as
> > > suggested by Andrey.
> > >
> > > Also, I still think that we should only resolve class loaders and not
> the
> > > class itself. The main reason is that Ignite will encode class names
> into
> > > an integer hash code and will store the integer->class-fqn mapping in
> > > internal replicated cache. I doubt users will get more efficient than
> an
> > > integer (4 bytes) for a class name.
> > >
> > > On the receiving side, once we are able to get the right class loader,
> we
> > > can easily get the proper class by calling
> > ClassLoader.findClass(class-fqn).
> > >
> > > Thoughts?
> > >
> > > D.
> > >
> > > On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <romain.gilles@gmail.com
> >
> > > wrote:
> > >
> > > > Hi,
> > > > Maybe a missing point. I think but I'm not sure that in the
> > > > OptimizedMarshaller there is a caching of already serialized classes.
> > Due
> > > > to the dynamic nature of OSGi this may lead to memory leak. In fact
> if
> > a
> > > > bundle is refreshed, it will produce a new BundleRevision and
> > therefore a
> > > > new classloader. And if you don't release the class from the previous
> > > > BundleRevision then you endup with memory leak. So maybe the
> Marshaller
> > > > interface or somewhere should provide a way to free those classes /
> > > > classloaders.
> > > >
> > > > Regards,
> > > >
> > > > Romain
> > > >
> > > > Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <andrewkornev@hotmail.com
> >
> > a
> > > > écrit :
> > > >
> > > > > Romain/Dmitriy,
> > > > >
> > > > > I prefer Romain's approach, but just curious, in the API you guys
> are
> > > > > proposing why use a byte[] rather than OutputStream/InputStream?
> > With a
> > > > > byte[], one would inevitably end up wrapping it into a byte stream
> > class.
> > > > > Also, the stream-based interface would be more in line with the
> > > > Marshaller
> > > > > API.
> > > > >
> > > > > Also for symmetry with the readClass() method, I suggest the
> > writeClass()
> > > > > take a Class<?> rather than an object.
> > > > >
> > > > > Regards
> > > > > Andrey
> > > > >
> > > > > > From: romain.gilles@gmail.com
> > > > > > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > To: dev@ignite.apache.org
> > > > > >
> > > > > > Hi Dmitriy,
> > > > > > I think your solution is good. Maybe I will change it a little
> > bit...
> > > > :P
> > > > > > I think you should delegate the Class resolution to the resolver.
> > > > Because
> > > > > > for example with your current solution the marshaller may before
> > (or
> > > > > after)
> > > > > > store the fqn of the class (maybe only the first time it
> encounter
> > it)
> > > > > but
> > > > > > in order to save the classloader context resolution the
> > implementation
> > > > of
> > > > > > the resolver may have to save the package name of the given
> object
> > and
> > > > > some
> > > > > > extra info therefore the content package name will be serialized
> > two
> > > > > times.
> > > > > > Ok, it's not a big deal.
> > > > > > But now if the resolver identify that it can reuse the same class
> > > > loader
> > > > > > for a couple of classes. It will be interesting for it to have a
> > > > > > serialization context in order to save, let say, classloader/id
> > mapping
> > > > > in
> > > > > > order to save the id instead of a more longer representation.
> > > > > > So I propose something like that:
> > > > > > *public interface ClassResolver {*
> > > > > > * // This method is invoked on the sender side to *
> > > > > > * // marshal the information about the class.*
> > > > > > * // where the context is a map style object that is reset/new
> > for
> > > > > each
> > > > > > object graph serialization.*
> > > > > > * public byte[] writeClass(Object o, Context context) throws
> > > > > > IgniteException;*
> > > > > >
> > > > > > * // This method is invoked on the receiving side to*
> > > > > > * // retrieve the class based on provided information.*
> > > > > > * // where the context is a map style object that is reset/new
> > for
> > > > > each
> > > > > > object graph serialization.*
> > > > > > * public Class<?> readClass(byte[], Context context) throws
> > > > > > IgniteException;*
> > > > > > *}*
> > > > > >
> > > > > > I think your proposal will solve our issue and maybe also open a
> > door
> > > > for
> > > > > > the osgi development.
> > > > > > Let me know what do you think about me proposal? :)
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Romain
> > > > > >
> > > > > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > > > a
> > > > > > écrit :
> > > > > >
> > > > > > > Romain,
> > > > > > >
> > > > > > > Can you comment on the ClassLoaderResolver suggestion that I
> > proposed
> > > > > > > earlier? Will it solve your problem?
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <
> > > > romain.gilles@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Raul,
> > > > > > > > - Do you plan to use the TCCL when marshalling in OSGi env?
> > If yes
> > > > > how?
> > > > > > > > - I like the point 2. Maybe for a graph of object that come
> > from
> > > > > > > different
> > > > > > > > packages / bundles you may have to recursively capture the
> > package
> > > > > > > version
> > > > > > > > of the individual element of your graph.
> > > > > > > > - For the point 3 I wonder if it will not over-complexify
> the
> > > > > solution.
> > > > > > > As
> > > > > > > > a developer you will have to think about it. And it is not
> > obvious
> > > > > in the
> > > > > > > > code where things are serialized or not. You may use lambda
> in
> > your
> > > > > > > > application code therefore the current package become what
> you
> > call
> > > > > a DTO
> > > > > > > > package. The current state of ignite does not enforce you to
> > > > specify
> > > > > it
> > > > > > > for
> > > > > > > > "classical" classloading application. If you introduce this
> > extra
> > > > > step
> > > > > > > for
> > > > > > > > OSGi ready application you will maybe discourage people to
> use
> > > > OSGi.
> > > > > > > >
> > > > > > > > To comeback to our solution. We start we a strong assumption:
> > our
> > > > > cluster
> > > > > > > > is homogeneous in term of code so, of course, it simplify our
> > life
> > > > > :).
> > > > > > > >
> > > > > > > > BTW if you can introduce an extension point in the class
> > resolution
> > > > > > > > mechanism it can be interesting for end user in order to
> allow
> > them
> > > > > to
> > > > > > > > optimize it based on there specific use cases.
> > > > > > > >
> > > > > > > > Romain.
> > > > > > > >
> > > > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <
> raul@evosent.com>
> > a
> > > > > écrit :
> > > > > > > >
> > > > > > > > > Hi Andrey,
> > > > > > > > >
> > > > > > > > > Thanks for the participation in this topic.
> > > > > > > > >
> > > > > > > > > I don't like the solution to incorporate the bundle
> symbolic
> > name
> > > > > in
> > > > > > > the
> > > > > > > > > serialised form. Nothing guarantees that the classdef will
> be
> > > > > located
> > > > > > > > under
> > > > > > > > > the same bundle in both source and target machines. We also
> > have
> > > > to
> > > > > > > take
> > > > > > > > > into account that OSGi allows for multiple versions of the
> > same
> > > > > > > > > bundle/packages to co-exist in the same container. So it
> > becomes
> > > > > more
> > > > > > > > > complex.
> > > > > > > > >
> > > > > > > > > Using the TCCL should work when serialising, but it'll
> > probably
> > > > be
> > > > > of
> > > > > > > no
> > > > > > > > > use when deserialising on the other end.
> > > > > > > > >
> > > > > > > > > I need to enhance Ignite to:
> > > > > > > > >
> > > > > > > > > 1. Use the TCCL when marshalling on one end.
> > > > > > > > > 2. Incorporate the package version of the class in the
> > serialised
> > > > > form
> > > > > > > if
> > > > > > > > > Ignite is running in an OSGi environment.
> > > > > > > > > 3. On the receiver end, discover cache entities / DTOs in
> all
> > > > > bundles
> > > > > > > > > through a custom OSGi manifest header or the like, as I
> > explained
> > > > > > > before.
> > > > > > > > > Package version must be taken into account.
> > > > > > > > >
> > > > > > > > > What do you think?
> > > > > > > > >
> > > > > > > > > Raúl.
> > > > > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <
> > andrewkornev@hotmail.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Raul,
> > > > > > > > > >
> > > > > > > > > > The fundamental hurdle we need to jump over to make
> Ignite
> > > > > > > OSGi-enabled
> > > > > > > > > is
> > > > > > > > > > the marshalling. More specifically the issue is with
> > > > > deserialization
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > classes that are provided by the bundles other than the
> > Ignite
> > > > > bundle
> > > > > > > > > > itself.
> > > > > > > > > >
> > > > > > > > > > When the Ignite transport layer receives a message it
> > needs to
> > > > > figure
> > > > > > > > out
> > > > > > > > > > how to deserialize the bytes and for that it needs to
> know
> > the
> > > > > bundle
> > > > > > > > > that
> > > > > > > > > > provides the class to be deserailized. At this point TCCL
> > is of
> > > > > no
> > > > > > > use.
> > > > > > > > > To
> > > > > > > > > > make things more complex, the class may contain other
> > classes
> > > > > that
> > > > > > > come
> > > > > > > > > > from other bundles, and so on recursively. This means
> that
> > each
> > > > > > > object
> > > > > > > > in
> > > > > > > > > > the hierarchy must be serialized with its bundle name (or
> > > > bundle
> > > > > id),
> > > > > > > > so
> > > > > > > > > > that the deserializer will then be able to correctly
> > resolve
> > > > the
> > > > > > > class
> > > > > > > > > > while traversing the object hierarchy during
> > deserialization.
> > > > > > > > > >
> > > > > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking
> the
> > > > > ability to
> > > > > > > > > plug
> > > > > > > > > > in a custom class resolver. Romain's solution was to use
> > Kryo
> > > > > that
> > > > > > > does
> > > > > > > > > > provide a way to customize class resolution. It has
> solved
> > the
> > > > > > > problem
> > > > > > > > of
> > > > > > > > > > capturing the bundle info and he was able to successfully
> > run
> > > > > Ignite
> > > > > > > > as a
> > > > > > > > > > bundle in an OSGi container (after some repackaging and
> > > > > inclusion of
> > > > > > > > the
> > > > > > > > > > manifest). But Kryo-based marshalling introduced a lot of
> > > > > complexity
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > code and incorrect use of Kryo's numerous serializers
> > caused
> > > > some
> > > > > > > weird
> > > > > > > > > > hard-to-debug issues in the Ignite core (like duplicate
> > cache
> > > > > entries
> > > > > > > > due
> > > > > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap
> > class --
> > > > > go
> > > > > > > > > > figure!). Overall the Kryo-based solution is brittle and
> > hard
> > > > to
> > > > > > > > > maintain.
> > > > > > > > > >
> > > > > > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > > > > > 1) enhance the OptimizedMarshaller to allow custom class
> > > > > resolution.
> > > > > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in
> > addition to
> > > > > the
> > > > > > > > > > original one) to be used in OSGi environment.
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > > Andrey
> > > > > > > > > >
> > > > > > > > > > > From: raulk@apache.org
> > > > > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > > > > > Subject: Re: OSGi migration may require a special
> > marshaller
> > > > > > > > > > > To: dev@ignite.apache.org
> > > > > > > > > > >
> > > > > > > > > > > Hi Romain,
> > > > > > > > > > >
> > > > > > > > > > > I'm working on the OSGi compatibility of Ignite. I
> > appreciate
> > > > > your
> > > > > > > > > input.
> > > > > > > > > > >
> > > > > > > > > > > I'm thinking about the situation you describe and I
> > wonder if
> > > > > > > you're
> > > > > > > > > > > exporting Ignite as an OSGi service which is then
> > consumed
> > > > from
> > > > > > > other
> > > > > > > > > > > bundles. Under this situation, it would be quite easy
> to
> > > > > reproduce
> > > > > > > > the
> > > > > > > > > > > behaviour you describe if Ignite is not resolving
> > classes via
> > > > > the
> > > > > > > > TCCL.
> > > > > > > > > > > Need to dig deeper into that.
> > > > > > > > > > >
> > > > > > > > > > > Off the top of my head, there are two alternatives to
> > solve
> > > > it:
> > > > > > > > > > >
> > > > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not
> > already
> > > > > > > used) –
> > > > > > > > > we
> > > > > > > > > > > gotta be wary of possible regressions.
> > > > > > > > > > > 2. Create a special OSGi header Ignite-Export-Package
> so
> > that
> > > > > > > bundles
> > > > > > > > > > > containing DTOs can expose packages to Ignite's
> > marshallers.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > >
> > > > > > > > > > > *Raúl Kripalani*
> > > > > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel |
> > Integration,
> > > > > Big
> > > > > > > Data
> > > > > > > > > and
> > > > > > > > > > > Messaging Engineer
> > > > > > > > > > > http://about.me/raulkripalani |
> > > > > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > > > > > romain.gilles@misys.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm really interested in this issue:
> > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 .
> We
> > > > some
> > > > > > > stuff
> > > > > > > > to
> > > > > > > > > > make
> > > > > > > > > > > > it work in our osgi environment. The main issue for
> us
> > now
> > > > > is the
> > > > > > > > > > > > serialization. I think it you will have to rework the
> > > > > > > > > > OptimizedMarshaller
> > > > > > > > > > > > or any other marshaller that works with object that
> > come
> > > > from
> > > > > > > > outside
> > > > > > > > > > your
> > > > > > > > > > > > class space.
> > > > > > > > > > > >
> > > > > > > > > > > > We have try kryo that works. Kryo provide an
> extension
> > > > point
> > > > > in
> > > > > > > > order
> > > > > > > > > > to
> > > > > > > > > > > > resolve the classes:
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > > > > > . With this extension we are able to solve the
> problem
> > of
> > > > > > > external
> > > > > > > > > > classes.
> > > > > > > > > > > > The only issue with kryo is that some classes need a
> > > > certain
> > > > > care
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > serialization process and therefore a specialized
> > > > serializer.
> > > > > > > > > > > >
> > > > > > > > > > > > So I would like to know from the community what do
> > think of
> > > > > > > > changing
> > > > > > > > > > the
> > > > > > > > > > > > way the optimized marshaller works or introducing the
> > > > > support of
> > > > > > > > yet
> > > > > > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks in advance,
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Romain.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > > > > > "Misys" is the trade name of the Misys group of
> > companies.
> > > > > This
> > > > > > > > email
> > > > > > > > > > and
> > > > > > > > > > > > any attachments have been scanned for known viruses
> > using
> > > > > > > multiple
> > > > > > > > > > > > scanners. This email message is intended for the
> named
> > > > > recipient
> > > > > > > > > only.
> > > > > > > > > > It
> > > > > > > > > > > > may be privileged and/or confidential. If you are not
> > the
> > > > > named
> > > > > > > > > > recipient
> > > > > > > > > > > > of this email please notify us immediately and do not
> > copy
> > > > > it or
> > > > > > > > use
> > > > > > > > > > it for
> > > > > > > > > > > > any purpose, nor disclose its contents to any other
> > person.
> > > > > This
> > > > > > > > > email
> > > > > > > > > > does
> > > > > > > > > > > > not constitute the commencement of legal relations
> > between
> > > > > you
> > > > > > > and
> > > > > > > > > > Misys.
> > > > > > > > > > > > Please refer to the executed contract between you and
> > the
> > > > > > > relevant
> > > > > > > > > > member
> > > > > > > > > > > > of the Misys group for the identity of the
> contracting
> > > > party
> > > > > with
> > > > > > > > > > which you
> > > > > > > > > > > > are dealing.
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Romain Gilles <ro...@gmail.com>.
Hi Raoul,
I'm totally agree with you! I think here there is a confusion around how to
solve marshalling for ignite in OSGi env:
1. We are talking about the potential opportunity of introducing an
interface in the marshalling framework in order to resolve class /
classloader.
2. Then there was a other topic on the implementation detail on how to
resolve class / classloader in an OSGi environment.
3. Finally there is a third topic on the usage of TCCL to handle the
marshalling.
The point 1 does not block anything as it is "just" a delegation on how to
resolve class / classloader. And it may help in OSGi
The point 2 is an implementation detail and verse badly the purpose. It is
up to the implementation regarding the use cases and the constraint they
have to follow the approach they want. I will say that Ignite must follow
the OSGi best practices. And if someone want to do it differently due to
some specific constraint it is up to them. And because we open it with the
point one then it should work.
The point 3 is an implementation strategy to be able to fix marshalling
issue. We need to keep in mind that we are not marshalling on data (DTO)
because Ignite is not only a Data Grid but also computation element against
the distributed data.
Does those points help into the discussion? For me only the first point
matter. If we solve the first point we will enable everything.
Romain.
Le mer. 4 nov. 2015 à 11:56, Raul Kripalani <ra...@evosent.com> a écrit :
> Could we rewind a little bit please? What is the point of using the bundle
> symbolic name in the first place?
>
> In OSGi, it is bad practice to refer to bundles directly. A bundle is a
> packaging unit, that's all, but it should not have any further
> consideration.
>
> It is perfectly feasible that a given package is exported by bundle A in a
> container, and by bundle B in another. One should not make any assumptions
> in that regard. A simple example of this is an enterprise-wide deployment
> of Ignite, where multiple apps in different OSGi containers share cache
> entities. Why would one expect these entities to be hosted in the same
> bundle across all apps?
>
> Referring to a bundle in OSGi world is like referring to a JAR in a normal
> Java environment. One would not always expect to resolve class A.B.C from
> bundle xyz-1.2.3.jar, right? You would resolve from the classpath.
>
> There are other mechanisms to reach into the OSGi environment and resolve a
> class arbitrarily from another bundle. Some of which I've pointed out. But
> relying on a bundle symbolic name receives an -1 from me.
>
> Can't elaborate now, I'm travelling.
>
> Will get back to my office tomorrow and I'll dig deeper into possibilities.
>
> Regards,
> Raúl.
> On 4 Nov 2015 03:07, "Dmitriy Setrakyan" <ds...@apache.org> wrote:
>
> > Andrey, et al,
> >
> > Can you provide a way on how to get a required Bundle object based on,
> say,
> > bundle symbolic name and version?
> >
> > I have reached the end of the internet trying to find a way in OSGI to
> look
> > up a Bundle based on something other than an actual Class belonging to
> that
> > bundle, but no luck.
> >
> > D.
> >
> > On Tue, Nov 3, 2015 at 5:15 PM, Andrey Kornev <an...@hotmail.com>
> > wrote:
> >
> > > Dmitriy,
> > >
> > > I think your approach will work, but I let Romain respond.
> > >
> > > Also, in terms of the implementation, please keep in mind that the
> > > resolver must be called for each non-JDK and non-Ignite core class (it
> > > would probably make sense to eschew storing class loaders for such
> > classes
> > > in favor of compactness of the serialized representation -- see below).
> > > Also, it's worth keeping a cache of already resolved class loaders per
> > > marshaller invocation (this is probably the context that Romain has
> > > mentioned in his previous posting) to minimize the number of the
> resolver
> > > calls.
> > >
> > > In terms of the resolver's implementation, the simplest way to
> serialize
> > > the class loader would be by capturing two pieces of information (both
> > > strings): the bundle symbolic name and the bundle version. This
> approach
> > > however may result in bloating of the serialized representation: I'd
> > > roughly estimate the overhead per element to be at least 20-30 bytes
> (the
> > > length of the symbolic name string, plus the length of the version
> > string).
> > > There are way to reduce the overhead (like serializing the hash code of
> > the
> > > bundle id string rather than the string itself, and then come up with a
> > > clever way of resolving the hash collisions), but they all come at cost
> > of
> > > increased complexity...
> > >
> > > An alternative approach would be rely on the special bundle id which is
> > an
> > > integer and is generated by the OSGi container. But in this case, all
> > nodes
> > > must ensure that all the bundles have consistent ids (bundle A with id
> 42
> > > on node N1, has the same id on every other node) which is not easy --
> > while
> > > not entirely impossible -- to guarantee. As long as the nodes are
> > > homogeneous (have the same set of bundles deployed) the OSGi container
> is
> > > guaranteed to assign to the bundles the same ids.
> > >
> > > Thanks
> > > Andrey
> > >
> > > > From: dsetrakyan@apache.org
> > > > Date: Tue, 3 Nov 2015 16:29:41 -0800
> > > > Subject: Re: OSGi migration may require a special marshaller
> > > > To: dev@ignite.apache.org
> > > >
> > > > Romain,
> > > >
> > > > In the upcoming release we will be deprecating the
> OptimizedMarshaller
> > > and
> > > > will be switching to a default internal marshaller (which is based on
> > the
> > > > new PortableMarshaller donated by GridGain).
> > > >
> > > > Having said that, we may be able to pass BinaryWriter and
> BinaryReader
> > > > instead of byte arrays. This will be pretty close to passing the
> > stream,
> > > as
> > > > suggested by Andrey.
> > > >
> > > > Also, I still think that we should only resolve class loaders and not
> > the
> > > > class itself. The main reason is that Ignite will encode class names
> > into
> > > > an integer hash code and will store the integer->class-fqn mapping in
> > > > internal replicated cache. I doubt users will get more efficient than
> > an
> > > > integer (4 bytes) for a class name.
> > > >
> > > > On the receiving side, once we are able to get the right class
> loader,
> > we
> > > > can easily get the proper class by calling
> > > ClassLoader.findClass(class-fqn).
> > > >
> > > > Thoughts?
> > > >
> > > > D.
> > > >
> > > > On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <
> romain.gilles@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > Maybe a missing point. I think but I'm not sure that in the
> > > > > OptimizedMarshaller there is a caching of already serialized
> classes.
> > > Due
> > > > > to the dynamic nature of OSGi this may lead to memory leak. In fact
> > if
> > > a
> > > > > bundle is refreshed, it will produce a new BundleRevision and
> > > therefore a
> > > > > new classloader. And if you don't release the class from the
> previous
> > > > > BundleRevision then you endup with memory leak. So maybe the
> > Marshaller
> > > > > interface or somewhere should provide a way to free those classes /
> > > > > classloaders.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Romain
> > > > >
> > > > > Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <
> andrewkornev@hotmail.com
> > >
> > > a
> > > > > écrit :
> > > > >
> > > > > > Romain/Dmitriy,
> > > > > >
> > > > > > I prefer Romain's approach, but just curious, in the API you guys
> > are
> > > > > > proposing why use a byte[] rather than OutputStream/InputStream?
> > > With a
> > > > > > byte[], one would inevitably end up wrapping it into a byte
> stream
> > > class.
> > > > > > Also, the stream-based interface would be more in line with the
> > > > > Marshaller
> > > > > > API.
> > > > > >
> > > > > > Also for symmetry with the readClass() method, I suggest the
> > > writeClass()
> > > > > > take a Class<?> rather than an object.
> > > > > >
> > > > > > Regards
> > > > > > Andrey
> > > > > >
> > > > > > > From: romain.gilles@gmail.com
> > > > > > > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > > To: dev@ignite.apache.org
> > > > > > >
> > > > > > > Hi Dmitriy,
> > > > > > > I think your solution is good. Maybe I will change it a little
> > > bit...
> > > > > :P
> > > > > > > I think you should delegate the Class resolution to the
> resolver.
> > > > > Because
> > > > > > > for example with your current solution the marshaller may
> before
> > > (or
> > > > > > after)
> > > > > > > store the fqn of the class (maybe only the first time it
> > encounter
> > > it)
> > > > > > but
> > > > > > > in order to save the classloader context resolution the
> > > implementation
> > > > > of
> > > > > > > the resolver may have to save the package name of the given
> > object
> > > and
> > > > > > some
> > > > > > > extra info therefore the content package name will be
> serialized
> > > two
> > > > > > times.
> > > > > > > Ok, it's not a big deal.
> > > > > > > But now if the resolver identify that it can reuse the same
> class
> > > > > loader
> > > > > > > for a couple of classes. It will be interesting for it to have
> a
> > > > > > > serialization context in order to save, let say, classloader/id
> > > mapping
> > > > > > in
> > > > > > > order to save the id instead of a more longer representation.
> > > > > > > So I propose something like that:
> > > > > > > *public interface ClassResolver {*
> > > > > > > * // This method is invoked on the sender side to *
> > > > > > > * // marshal the information about the class.*
> > > > > > > * // where the context is a map style object that is
> reset/new
> > > for
> > > > > > each
> > > > > > > object graph serialization.*
> > > > > > > * public byte[] writeClass(Object o, Context context) throws
> > > > > > > IgniteException;*
> > > > > > >
> > > > > > > * // This method is invoked on the receiving side to*
> > > > > > > * // retrieve the class based on provided information.*
> > > > > > > * // where the context is a map style object that is
> reset/new
> > > for
> > > > > > each
> > > > > > > object graph serialization.*
> > > > > > > * public Class<?> readClass(byte[], Context context) throws
> > > > > > > IgniteException;*
> > > > > > > *}*
> > > > > > >
> > > > > > > I think your proposal will solve our issue and maybe also open
> a
> > > door
> > > > > for
> > > > > > > the osgi development.
> > > > > > > Let me know what do you think about me proposal? :)
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Romain
> > > > > > >
> > > > > > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org>
> > > > > a
> > > > > > > écrit :
> > > > > > >
> > > > > > > > Romain,
> > > > > > > >
> > > > > > > > Can you comment on the ClassLoaderResolver suggestion that I
> > > proposed
> > > > > > > > earlier? Will it solve your problem?
> > > > > > > >
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <
> > > > > romain.gilles@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Raul,
> > > > > > > > > - Do you plan to use the TCCL when marshalling in OSGi
> env?
> > > If yes
> > > > > > how?
> > > > > > > > > - I like the point 2. Maybe for a graph of object that
> come
> > > from
> > > > > > > > different
> > > > > > > > > packages / bundles you may have to recursively capture the
> > > package
> > > > > > > > version
> > > > > > > > > of the individual element of your graph.
> > > > > > > > > - For the point 3 I wonder if it will not over-complexify
> > the
> > > > > > solution.
> > > > > > > > As
> > > > > > > > > a developer you will have to think about it. And it is not
> > > obvious
> > > > > > in the
> > > > > > > > > code where things are serialized or not. You may use lambda
> > in
> > > your
> > > > > > > > > application code therefore the current package become what
> > you
> > > call
> > > > > > a DTO
> > > > > > > > > package. The current state of ignite does not enforce you
> to
> > > > > specify
> > > > > > it
> > > > > > > > for
> > > > > > > > > "classical" classloading application. If you introduce this
> > > extra
> > > > > > step
> > > > > > > > for
> > > > > > > > > OSGi ready application you will maybe discourage people to
> > use
> > > > > OSGi.
> > > > > > > > >
> > > > > > > > > To comeback to our solution. We start we a strong
> assumption:
> > > our
> > > > > > cluster
> > > > > > > > > is homogeneous in term of code so, of course, it simplify
> our
> > > life
> > > > > > :).
> > > > > > > > >
> > > > > > > > > BTW if you can introduce an extension point in the class
> > > resolution
> > > > > > > > > mechanism it can be interesting for end user in order to
> > allow
> > > them
> > > > > > to
> > > > > > > > > optimize it based on there specific use cases.
> > > > > > > > >
> > > > > > > > > Romain.
> > > > > > > > >
> > > > > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <
> > raul@evosent.com>
> > > a
> > > > > > écrit :
> > > > > > > > >
> > > > > > > > > > Hi Andrey,
> > > > > > > > > >
> > > > > > > > > > Thanks for the participation in this topic.
> > > > > > > > > >
> > > > > > > > > > I don't like the solution to incorporate the bundle
> > symbolic
> > > name
> > > > > > in
> > > > > > > > the
> > > > > > > > > > serialised form. Nothing guarantees that the classdef
> will
> > be
> > > > > > located
> > > > > > > > > under
> > > > > > > > > > the same bundle in both source and target machines. We
> also
> > > have
> > > > > to
> > > > > > > > take
> > > > > > > > > > into account that OSGi allows for multiple versions of
> the
> > > same
> > > > > > > > > > bundle/packages to co-exist in the same container. So it
> > > becomes
> > > > > > more
> > > > > > > > > > complex.
> > > > > > > > > >
> > > > > > > > > > Using the TCCL should work when serialising, but it'll
> > > probably
> > > > > be
> > > > > > of
> > > > > > > > no
> > > > > > > > > > use when deserialising on the other end.
> > > > > > > > > >
> > > > > > > > > > I need to enhance Ignite to:
> > > > > > > > > >
> > > > > > > > > > 1. Use the TCCL when marshalling on one end.
> > > > > > > > > > 2. Incorporate the package version of the class in the
> > > serialised
> > > > > > form
> > > > > > > > if
> > > > > > > > > > Ignite is running in an OSGi environment.
> > > > > > > > > > 3. On the receiver end, discover cache entities / DTOs in
> > all
> > > > > > bundles
> > > > > > > > > > through a custom OSGi manifest header or the like, as I
> > > explained
> > > > > > > > before.
> > > > > > > > > > Package version must be taken into account.
> > > > > > > > > >
> > > > > > > > > > What do you think?
> > > > > > > > > >
> > > > > > > > > > Raúl.
> > > > > > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <
> > > andrewkornev@hotmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Raul,
> > > > > > > > > > >
> > > > > > > > > > > The fundamental hurdle we need to jump over to make
> > Ignite
> > > > > > > > OSGi-enabled
> > > > > > > > > > is
> > > > > > > > > > > the marshalling. More specifically the issue is with
> > > > > > deserialization
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > classes that are provided by the bundles other than the
> > > Ignite
> > > > > > bundle
> > > > > > > > > > > itself.
> > > > > > > > > > >
> > > > > > > > > > > When the Ignite transport layer receives a message it
> > > needs to
> > > > > > figure
> > > > > > > > > out
> > > > > > > > > > > how to deserialize the bytes and for that it needs to
> > know
> > > the
> > > > > > bundle
> > > > > > > > > > that
> > > > > > > > > > > provides the class to be deserailized. At this point
> TCCL
> > > is of
> > > > > > no
> > > > > > > > use.
> > > > > > > > > > To
> > > > > > > > > > > make things more complex, the class may contain other
> > > classes
> > > > > > that
> > > > > > > > come
> > > > > > > > > > > from other bundles, and so on recursively. This means
> > that
> > > each
> > > > > > > > object
> > > > > > > > > in
> > > > > > > > > > > the hierarchy must be serialized with its bundle name
> (or
> > > > > bundle
> > > > > > id),
> > > > > > > > > so
> > > > > > > > > > > that the deserializer will then be able to correctly
> > > resolve
> > > > > the
> > > > > > > > class
> > > > > > > > > > > while traversing the object hierarchy during
> > > deserialization.
> > > > > > > > > > >
> > > > > > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking
> > the
> > > > > > ability to
> > > > > > > > > > plug
> > > > > > > > > > > in a custom class resolver. Romain's solution was to
> use
> > > Kryo
> > > > > > that
> > > > > > > > does
> > > > > > > > > > > provide a way to customize class resolution. It has
> > solved
> > > the
> > > > > > > > problem
> > > > > > > > > of
> > > > > > > > > > > capturing the bundle info and he was able to
> successfully
> > > run
> > > > > > Ignite
> > > > > > > > > as a
> > > > > > > > > > > bundle in an OSGi container (after some repackaging and
> > > > > > inclusion of
> > > > > > > > > the
> > > > > > > > > > > manifest). But Kryo-based marshalling introduced a lot
> of
> > > > > > complexity
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > code and incorrect use of Kryo's numerous serializers
> > > caused
> > > > > some
> > > > > > > > weird
> > > > > > > > > > > hard-to-debug issues in the Ignite core (like duplicate
> > > cache
> > > > > > entries
> > > > > > > > > due
> > > > > > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap
> > > class --
> > > > > > go
> > > > > > > > > > > figure!). Overall the Kryo-based solution is brittle
> and
> > > hard
> > > > > to
> > > > > > > > > > maintain.
> > > > > > > > > > >
> > > > > > > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > > > > > > 1) enhance the OptimizedMarshaller to allow custom
> class
> > > > > > resolution.
> > > > > > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in
> > > addition to
> > > > > > the
> > > > > > > > > > > original one) to be used in OSGi environment.
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > > Andrey
> > > > > > > > > > >
> > > > > > > > > > > > From: raulk@apache.org
> > > > > > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > > > > > > Subject: Re: OSGi migration may require a special
> > > marshaller
> > > > > > > > > > > > To: dev@ignite.apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Romain,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm working on the OSGi compatibility of Ignite. I
> > > appreciate
> > > > > > your
> > > > > > > > > > input.
> > > > > > > > > > > >
> > > > > > > > > > > > I'm thinking about the situation you describe and I
> > > wonder if
> > > > > > > > you're
> > > > > > > > > > > > exporting Ignite as an OSGi service which is then
> > > consumed
> > > > > from
> > > > > > > > other
> > > > > > > > > > > > bundles. Under this situation, it would be quite easy
> > to
> > > > > > reproduce
> > > > > > > > > the
> > > > > > > > > > > > behaviour you describe if Ignite is not resolving
> > > classes via
> > > > > > the
> > > > > > > > > TCCL.
> > > > > > > > > > > > Need to dig deeper into that.
> > > > > > > > > > > >
> > > > > > > > > > > > Off the top of my head, there are two alternatives to
> > > solve
> > > > > it:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not
> > > already
> > > > > > > > used) –
> > > > > > > > > > we
> > > > > > > > > > > > gotta be wary of possible regressions.
> > > > > > > > > > > > 2. Create a special OSGi header Ignite-Export-Package
> > so
> > > that
> > > > > > > > bundles
> > > > > > > > > > > > containing DTOs can expose packages to Ignite's
> > > marshallers.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > >
> > > > > > > > > > > > *Raúl Kripalani*
> > > > > > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel |
> > > Integration,
> > > > > > Big
> > > > > > > > Data
> > > > > > > > > > and
> > > > > > > > > > > > Messaging Engineer
> > > > > > > > > > > > http://about.me/raulkripalani |
> > > > > > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > > > > > > romain.gilles@misys.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm really interested in this issue:
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270
> .
> > We
> > > > > some
> > > > > > > > stuff
> > > > > > > > > to
> > > > > > > > > > > make
> > > > > > > > > > > > > it work in our osgi environment. The main issue for
> > us
> > > now
> > > > > > is the
> > > > > > > > > > > > > serialization. I think it you will have to rework
> the
> > > > > > > > > > > OptimizedMarshaller
> > > > > > > > > > > > > or any other marshaller that works with object that
> > > come
> > > > > from
> > > > > > > > > outside
> > > > > > > > > > > your
> > > > > > > > > > > > > class space.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We have try kryo that works. Kryo provide an
> > extension
> > > > > point
> > > > > > in
> > > > > > > > > order
> > > > > > > > > > > to
> > > > > > > > > > > > > resolve the classes:
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > > > > > > . With this extension we are able to solve the
> > problem
> > > of
> > > > > > > > external
> > > > > > > > > > > classes.
> > > > > > > > > > > > > The only issue with kryo is that some classes need
> a
> > > > > certain
> > > > > > care
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > serialization process and therefore a specialized
> > > > > serializer.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So I would like to know from the community what do
> > > think of
> > > > > > > > > changing
> > > > > > > > > > > the
> > > > > > > > > > > > > way the optimized marshaller works or introducing
> the
> > > > > > support of
> > > > > > > > > yet
> > > > > > > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks in advance,
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Romain.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > > > > > > "Misys" is the trade name of the Misys group of
> > > companies.
> > > > > > This
> > > > > > > > > email
> > > > > > > > > > > and
> > > > > > > > > > > > > any attachments have been scanned for known viruses
> > > using
> > > > > > > > multiple
> > > > > > > > > > > > > scanners. This email message is intended for the
> > named
> > > > > > recipient
> > > > > > > > > > only.
> > > > > > > > > > > It
> > > > > > > > > > > > > may be privileged and/or confidential. If you are
> not
> > > the
> > > > > > named
> > > > > > > > > > > recipient
> > > > > > > > > > > > > of this email please notify us immediately and do
> not
> > > copy
> > > > > > it or
> > > > > > > > > use
> > > > > > > > > > > it for
> > > > > > > > > > > > > any purpose, nor disclose its contents to any other
> > > person.
> > > > > > This
> > > > > > > > > > email
> > > > > > > > > > > does
> > > > > > > > > > > > > not constitute the commencement of legal relations
> > > between
> > > > > > you
> > > > > > > > and
> > > > > > > > > > > Misys.
> > > > > > > > > > > > > Please refer to the executed contract between you
> and
> > > the
> > > > > > > > relevant
> > > > > > > > > > > member
> > > > > > > > > > > > > of the Misys group for the identity of the
> > contracting
> > > > > party
> > > > > > with
> > > > > > > > > > > which you
> > > > > > > > > > > > > are dealing.
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> > >
> >
>
RE: OSGi migration may require a special marshaller
Posted by Andrey Kornev <an...@hotmail.com>.
Raul,
While we're waiting for the specific proposal(s) from you, I'd like to point out that OSGi bundle is much more than just a jar or a packaging unit. First and foremost an OSGi bundle is also a class loader unlike the jars in non-OSGi environment. We're after the class loader hence the proposal to use the bundle symbolic name and bundle version for serialization of the *class loader*.
Have a safe trip and please, do get back to us at your earliest.
Regards
Andrey
> Date: Wed, 4 Nov 2015 10:56:32 +0000
> Subject: Re: OSGi migration may require a special marshaller
> From: raul@evosent.com
> To: dev@ignite.apache.org
>
> Could we rewind a little bit please? What is the point of using the bundle
> symbolic name in the first place?
>
> In OSGi, it is bad practice to refer to bundles directly. A bundle is a
> packaging unit, that's all, but it should not have any further
> consideration.
>
> It is perfectly feasible that a given package is exported by bundle A in a
> container, and by bundle B in another. One should not make any assumptions
> in that regard. A simple example of this is an enterprise-wide deployment
> of Ignite, where multiple apps in different OSGi containers share cache
> entities. Why would one expect these entities to be hosted in the same
> bundle across all apps?
>
> Referring to a bundle in OSGi world is like referring to a JAR in a normal
> Java environment. One would not always expect to resolve class A.B.C from
> bundle xyz-1.2.3.jar, right? You would resolve from the classpath.
>
> There are other mechanisms to reach into the OSGi environment and resolve a
> class arbitrarily from another bundle. Some of which I've pointed out. But
> relying on a bundle symbolic name receives an -1 from me.
>
> Can't elaborate now, I'm travelling.
>
> Will get back to my office tomorrow and I'll dig deeper into possibilities.
>
> Regards,
> Raúl.
> On 4 Nov 2015 03:07, "Dmitriy Setrakyan" <ds...@apache.org> wrote:
>
> > Andrey, et al,
> >
> > Can you provide a way on how to get a required Bundle object based on, say,
> > bundle symbolic name and version?
> >
> > I have reached the end of the internet trying to find a way in OSGI to look
> > up a Bundle based on something other than an actual Class belonging to that
> > bundle, but no luck.
> >
> > D.
> >
> > On Tue, Nov 3, 2015 at 5:15 PM, Andrey Kornev <an...@hotmail.com>
> > wrote:
> >
> > > Dmitriy,
> > >
> > > I think your approach will work, but I let Romain respond.
> > >
> > > Also, in terms of the implementation, please keep in mind that the
> > > resolver must be called for each non-JDK and non-Ignite core class (it
> > > would probably make sense to eschew storing class loaders for such
> > classes
> > > in favor of compactness of the serialized representation -- see below).
> > > Also, it's worth keeping a cache of already resolved class loaders per
> > > marshaller invocation (this is probably the context that Romain has
> > > mentioned in his previous posting) to minimize the number of the resolver
> > > calls.
> > >
> > > In terms of the resolver's implementation, the simplest way to serialize
> > > the class loader would be by capturing two pieces of information (both
> > > strings): the bundle symbolic name and the bundle version. This approach
> > > however may result in bloating of the serialized representation: I'd
> > > roughly estimate the overhead per element to be at least 20-30 bytes (the
> > > length of the symbolic name string, plus the length of the version
> > string).
> > > There are way to reduce the overhead (like serializing the hash code of
> > the
> > > bundle id string rather than the string itself, and then come up with a
> > > clever way of resolving the hash collisions), but they all come at cost
> > of
> > > increased complexity...
> > >
> > > An alternative approach would be rely on the special bundle id which is
> > an
> > > integer and is generated by the OSGi container. But in this case, all
> > nodes
> > > must ensure that all the bundles have consistent ids (bundle A with id 42
> > > on node N1, has the same id on every other node) which is not easy --
> > while
> > > not entirely impossible -- to guarantee. As long as the nodes are
> > > homogeneous (have the same set of bundles deployed) the OSGi container is
> > > guaranteed to assign to the bundles the same ids.
> > >
> > > Thanks
> > > Andrey
> > >
> > > > From: dsetrakyan@apache.org
> > > > Date: Tue, 3 Nov 2015 16:29:41 -0800
> > > > Subject: Re: OSGi migration may require a special marshaller
> > > > To: dev@ignite.apache.org
> > > >
> > > > Romain,
> > > >
> > > > In the upcoming release we will be deprecating the OptimizedMarshaller
> > > and
> > > > will be switching to a default internal marshaller (which is based on
> > the
> > > > new PortableMarshaller donated by GridGain).
> > > >
> > > > Having said that, we may be able to pass BinaryWriter and BinaryReader
> > > > instead of byte arrays. This will be pretty close to passing the
> > stream,
> > > as
> > > > suggested by Andrey.
> > > >
> > > > Also, I still think that we should only resolve class loaders and not
> > the
> > > > class itself. The main reason is that Ignite will encode class names
> > into
> > > > an integer hash code and will store the integer->class-fqn mapping in
> > > > internal replicated cache. I doubt users will get more efficient than
> > an
> > > > integer (4 bytes) for a class name.
> > > >
> > > > On the receiving side, once we are able to get the right class loader,
> > we
> > > > can easily get the proper class by calling
> > > ClassLoader.findClass(class-fqn).
> > > >
> > > > Thoughts?
> > > >
> > > > D.
> > > >
> > > > On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <romain.gilles@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > Maybe a missing point. I think but I'm not sure that in the
> > > > > OptimizedMarshaller there is a caching of already serialized classes.
> > > Due
> > > > > to the dynamic nature of OSGi this may lead to memory leak. In fact
> > if
> > > a
> > > > > bundle is refreshed, it will produce a new BundleRevision and
> > > therefore a
> > > > > new classloader. And if you don't release the class from the previous
> > > > > BundleRevision then you endup with memory leak. So maybe the
> > Marshaller
> > > > > interface or somewhere should provide a way to free those classes /
> > > > > classloaders.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Romain
> > > > >
> > > > > Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <andrewkornev@hotmail.com
> > >
> > > a
> > > > > écrit :
> > > > >
> > > > > > Romain/Dmitriy,
> > > > > >
> > > > > > I prefer Romain's approach, but just curious, in the API you guys
> > are
> > > > > > proposing why use a byte[] rather than OutputStream/InputStream?
> > > With a
> > > > > > byte[], one would inevitably end up wrapping it into a byte stream
> > > class.
> > > > > > Also, the stream-based interface would be more in line with the
> > > > > Marshaller
> > > > > > API.
> > > > > >
> > > > > > Also for symmetry with the readClass() method, I suggest the
> > > writeClass()
> > > > > > take a Class<?> rather than an object.
> > > > > >
> > > > > > Regards
> > > > > > Andrey
> > > > > >
> > > > > > > From: romain.gilles@gmail.com
> > > > > > > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > > To: dev@ignite.apache.org
> > > > > > >
> > > > > > > Hi Dmitriy,
> > > > > > > I think your solution is good. Maybe I will change it a little
> > > bit...
> > > > > :P
> > > > > > > I think you should delegate the Class resolution to the resolver.
> > > > > Because
> > > > > > > for example with your current solution the marshaller may before
> > > (or
> > > > > > after)
> > > > > > > store the fqn of the class (maybe only the first time it
> > encounter
> > > it)
> > > > > > but
> > > > > > > in order to save the classloader context resolution the
> > > implementation
> > > > > of
> > > > > > > the resolver may have to save the package name of the given
> > object
> > > and
> > > > > > some
> > > > > > > extra info therefore the content package name will be serialized
> > > two
> > > > > > times.
> > > > > > > Ok, it's not a big deal.
> > > > > > > But now if the resolver identify that it can reuse the same class
> > > > > loader
> > > > > > > for a couple of classes. It will be interesting for it to have a
> > > > > > > serialization context in order to save, let say, classloader/id
> > > mapping
> > > > > > in
> > > > > > > order to save the id instead of a more longer representation.
> > > > > > > So I propose something like that:
> > > > > > > *public interface ClassResolver {*
> > > > > > > * // This method is invoked on the sender side to *
> > > > > > > * // marshal the information about the class.*
> > > > > > > * // where the context is a map style object that is reset/new
> > > for
> > > > > > each
> > > > > > > object graph serialization.*
> > > > > > > * public byte[] writeClass(Object o, Context context) throws
> > > > > > > IgniteException;*
> > > > > > >
> > > > > > > * // This method is invoked on the receiving side to*
> > > > > > > * // retrieve the class based on provided information.*
> > > > > > > * // where the context is a map style object that is reset/new
> > > for
> > > > > > each
> > > > > > > object graph serialization.*
> > > > > > > * public Class<?> readClass(byte[], Context context) throws
> > > > > > > IgniteException;*
> > > > > > > *}*
> > > > > > >
> > > > > > > I think your proposal will solve our issue and maybe also open a
> > > door
> > > > > for
> > > > > > > the osgi development.
> > > > > > > Let me know what do you think about me proposal? :)
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Romain
> > > > > > >
> > > > > > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org>
> > > > > a
> > > > > > > écrit :
> > > > > > >
> > > > > > > > Romain,
> > > > > > > >
> > > > > > > > Can you comment on the ClassLoaderResolver suggestion that I
> > > proposed
> > > > > > > > earlier? Will it solve your problem?
> > > > > > > >
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <
> > > > > romain.gilles@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Raul,
> > > > > > > > > - Do you plan to use the TCCL when marshalling in OSGi env?
> > > If yes
> > > > > > how?
> > > > > > > > > - I like the point 2. Maybe for a graph of object that come
> > > from
> > > > > > > > different
> > > > > > > > > packages / bundles you may have to recursively capture the
> > > package
> > > > > > > > version
> > > > > > > > > of the individual element of your graph.
> > > > > > > > > - For the point 3 I wonder if it will not over-complexify
> > the
> > > > > > solution.
> > > > > > > > As
> > > > > > > > > a developer you will have to think about it. And it is not
> > > obvious
> > > > > > in the
> > > > > > > > > code where things are serialized or not. You may use lambda
> > in
> > > your
> > > > > > > > > application code therefore the current package become what
> > you
> > > call
> > > > > > a DTO
> > > > > > > > > package. The current state of ignite does not enforce you to
> > > > > specify
> > > > > > it
> > > > > > > > for
> > > > > > > > > "classical" classloading application. If you introduce this
> > > extra
> > > > > > step
> > > > > > > > for
> > > > > > > > > OSGi ready application you will maybe discourage people to
> > use
> > > > > OSGi.
> > > > > > > > >
> > > > > > > > > To comeback to our solution. We start we a strong assumption:
> > > our
> > > > > > cluster
> > > > > > > > > is homogeneous in term of code so, of course, it simplify our
> > > life
> > > > > > :).
> > > > > > > > >
> > > > > > > > > BTW if you can introduce an extension point in the class
> > > resolution
> > > > > > > > > mechanism it can be interesting for end user in order to
> > allow
> > > them
> > > > > > to
> > > > > > > > > optimize it based on there specific use cases.
> > > > > > > > >
> > > > > > > > > Romain.
> > > > > > > > >
> > > > > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <
> > raul@evosent.com>
> > > a
> > > > > > écrit :
> > > > > > > > >
> > > > > > > > > > Hi Andrey,
> > > > > > > > > >
> > > > > > > > > > Thanks for the participation in this topic.
> > > > > > > > > >
> > > > > > > > > > I don't like the solution to incorporate the bundle
> > symbolic
> > > name
> > > > > > in
> > > > > > > > the
> > > > > > > > > > serialised form. Nothing guarantees that the classdef will
> > be
> > > > > > located
> > > > > > > > > under
> > > > > > > > > > the same bundle in both source and target machines. We also
> > > have
> > > > > to
> > > > > > > > take
> > > > > > > > > > into account that OSGi allows for multiple versions of the
> > > same
> > > > > > > > > > bundle/packages to co-exist in the same container. So it
> > > becomes
> > > > > > more
> > > > > > > > > > complex.
> > > > > > > > > >
> > > > > > > > > > Using the TCCL should work when serialising, but it'll
> > > probably
> > > > > be
> > > > > > of
> > > > > > > > no
> > > > > > > > > > use when deserialising on the other end.
> > > > > > > > > >
> > > > > > > > > > I need to enhance Ignite to:
> > > > > > > > > >
> > > > > > > > > > 1. Use the TCCL when marshalling on one end.
> > > > > > > > > > 2. Incorporate the package version of the class in the
> > > serialised
> > > > > > form
> > > > > > > > if
> > > > > > > > > > Ignite is running in an OSGi environment.
> > > > > > > > > > 3. On the receiver end, discover cache entities / DTOs in
> > all
> > > > > > bundles
> > > > > > > > > > through a custom OSGi manifest header or the like, as I
> > > explained
> > > > > > > > before.
> > > > > > > > > > Package version must be taken into account.
> > > > > > > > > >
> > > > > > > > > > What do you think?
> > > > > > > > > >
> > > > > > > > > > Raúl.
> > > > > > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <
> > > andrewkornev@hotmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Raul,
> > > > > > > > > > >
> > > > > > > > > > > The fundamental hurdle we need to jump over to make
> > Ignite
> > > > > > > > OSGi-enabled
> > > > > > > > > > is
> > > > > > > > > > > the marshalling. More specifically the issue is with
> > > > > > deserialization
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > classes that are provided by the bundles other than the
> > > Ignite
> > > > > > bundle
> > > > > > > > > > > itself.
> > > > > > > > > > >
> > > > > > > > > > > When the Ignite transport layer receives a message it
> > > needs to
> > > > > > figure
> > > > > > > > > out
> > > > > > > > > > > how to deserialize the bytes and for that it needs to
> > know
> > > the
> > > > > > bundle
> > > > > > > > > > that
> > > > > > > > > > > provides the class to be deserailized. At this point TCCL
> > > is of
> > > > > > no
> > > > > > > > use.
> > > > > > > > > > To
> > > > > > > > > > > make things more complex, the class may contain other
> > > classes
> > > > > > that
> > > > > > > > come
> > > > > > > > > > > from other bundles, and so on recursively. This means
> > that
> > > each
> > > > > > > > object
> > > > > > > > > in
> > > > > > > > > > > the hierarchy must be serialized with its bundle name (or
> > > > > bundle
> > > > > > id),
> > > > > > > > > so
> > > > > > > > > > > that the deserializer will then be able to correctly
> > > resolve
> > > > > the
> > > > > > > > class
> > > > > > > > > > > while traversing the object hierarchy during
> > > deserialization.
> > > > > > > > > > >
> > > > > > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking
> > the
> > > > > > ability to
> > > > > > > > > > plug
> > > > > > > > > > > in a custom class resolver. Romain's solution was to use
> > > Kryo
> > > > > > that
> > > > > > > > does
> > > > > > > > > > > provide a way to customize class resolution. It has
> > solved
> > > the
> > > > > > > > problem
> > > > > > > > > of
> > > > > > > > > > > capturing the bundle info and he was able to successfully
> > > run
> > > > > > Ignite
> > > > > > > > > as a
> > > > > > > > > > > bundle in an OSGi container (after some repackaging and
> > > > > > inclusion of
> > > > > > > > > the
> > > > > > > > > > > manifest). But Kryo-based marshalling introduced a lot of
> > > > > > complexity
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > code and incorrect use of Kryo's numerous serializers
> > > caused
> > > > > some
> > > > > > > > weird
> > > > > > > > > > > hard-to-debug issues in the Ignite core (like duplicate
> > > cache
> > > > > > entries
> > > > > > > > > due
> > > > > > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap
> > > class --
> > > > > > go
> > > > > > > > > > > figure!). Overall the Kryo-based solution is brittle and
> > > hard
> > > > > to
> > > > > > > > > > maintain.
> > > > > > > > > > >
> > > > > > > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > > > > > > 1) enhance the OptimizedMarshaller to allow custom class
> > > > > > resolution.
> > > > > > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in
> > > addition to
> > > > > > the
> > > > > > > > > > > original one) to be used in OSGi environment.
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > > Andrey
> > > > > > > > > > >
> > > > > > > > > > > > From: raulk@apache.org
> > > > > > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > > > > > > Subject: Re: OSGi migration may require a special
> > > marshaller
> > > > > > > > > > > > To: dev@ignite.apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Romain,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm working on the OSGi compatibility of Ignite. I
> > > appreciate
> > > > > > your
> > > > > > > > > > input.
> > > > > > > > > > > >
> > > > > > > > > > > > I'm thinking about the situation you describe and I
> > > wonder if
> > > > > > > > you're
> > > > > > > > > > > > exporting Ignite as an OSGi service which is then
> > > consumed
> > > > > from
> > > > > > > > other
> > > > > > > > > > > > bundles. Under this situation, it would be quite easy
> > to
> > > > > > reproduce
> > > > > > > > > the
> > > > > > > > > > > > behaviour you describe if Ignite is not resolving
> > > classes via
> > > > > > the
> > > > > > > > > TCCL.
> > > > > > > > > > > > Need to dig deeper into that.
> > > > > > > > > > > >
> > > > > > > > > > > > Off the top of my head, there are two alternatives to
> > > solve
> > > > > it:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not
> > > already
> > > > > > > > used) –
> > > > > > > > > > we
> > > > > > > > > > > > gotta be wary of possible regressions.
> > > > > > > > > > > > 2. Create a special OSGi header Ignite-Export-Package
> > so
> > > that
> > > > > > > > bundles
> > > > > > > > > > > > containing DTOs can expose packages to Ignite's
> > > marshallers.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > >
> > > > > > > > > > > > *Raúl Kripalani*
> > > > > > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel |
> > > Integration,
> > > > > > Big
> > > > > > > > Data
> > > > > > > > > > and
> > > > > > > > > > > > Messaging Engineer
> > > > > > > > > > > > http://about.me/raulkripalani |
> > > > > > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > > > > > > romain.gilles@misys.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm really interested in this issue:
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 .
> > We
> > > > > some
> > > > > > > > stuff
> > > > > > > > > to
> > > > > > > > > > > make
> > > > > > > > > > > > > it work in our osgi environment. The main issue for
> > us
> > > now
> > > > > > is the
> > > > > > > > > > > > > serialization. I think it you will have to rework the
> > > > > > > > > > > OptimizedMarshaller
> > > > > > > > > > > > > or any other marshaller that works with object that
> > > come
> > > > > from
> > > > > > > > > outside
> > > > > > > > > > > your
> > > > > > > > > > > > > class space.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We have try kryo that works. Kryo provide an
> > extension
> > > > > point
> > > > > > in
> > > > > > > > > order
> > > > > > > > > > > to
> > > > > > > > > > > > > resolve the classes:
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> > https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > > > > > > . With this extension we are able to solve the
> > problem
> > > of
> > > > > > > > external
> > > > > > > > > > > classes.
> > > > > > > > > > > > > The only issue with kryo is that some classes need a
> > > > > certain
> > > > > > care
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > serialization process and therefore a specialized
> > > > > serializer.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So I would like to know from the community what do
> > > think of
> > > > > > > > > changing
> > > > > > > > > > > the
> > > > > > > > > > > > > way the optimized marshaller works or introducing the
> > > > > > support of
> > > > > > > > > yet
> > > > > > > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks in advance,
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Romain.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > > > > > > "Misys" is the trade name of the Misys group of
> > > companies.
> > > > > > This
> > > > > > > > > email
> > > > > > > > > > > and
> > > > > > > > > > > > > any attachments have been scanned for known viruses
> > > using
> > > > > > > > multiple
> > > > > > > > > > > > > scanners. This email message is intended for the
> > named
> > > > > > recipient
> > > > > > > > > > only.
> > > > > > > > > > > It
> > > > > > > > > > > > > may be privileged and/or confidential. If you are not
> > > the
> > > > > > named
> > > > > > > > > > > recipient
> > > > > > > > > > > > > of this email please notify us immediately and do not
> > > copy
> > > > > > it or
> > > > > > > > > use
> > > > > > > > > > > it for
> > > > > > > > > > > > > any purpose, nor disclose its contents to any other
> > > person.
> > > > > > This
> > > > > > > > > > email
> > > > > > > > > > > does
> > > > > > > > > > > > > not constitute the commencement of legal relations
> > > between
> > > > > > you
> > > > > > > > and
> > > > > > > > > > > Misys.
> > > > > > > > > > > > > Please refer to the executed contract between you and
> > > the
> > > > > > > > relevant
> > > > > > > > > > > member
> > > > > > > > > > > > > of the Misys group for the identity of the
> > contracting
> > > > > party
> > > > > > with
> > > > > > > > > > > which you
> > > > > > > > > > > > > are dealing.
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> > >
> >
Re: OSGi migration may require a special marshaller
Posted by Raul Kripalani <ra...@evosent.com>.
Could we rewind a little bit please? What is the point of using the bundle
symbolic name in the first place?
In OSGi, it is bad practice to refer to bundles directly. A bundle is a
packaging unit, that's all, but it should not have any further
consideration.
It is perfectly feasible that a given package is exported by bundle A in a
container, and by bundle B in another. One should not make any assumptions
in that regard. A simple example of this is an enterprise-wide deployment
of Ignite, where multiple apps in different OSGi containers share cache
entities. Why would one expect these entities to be hosted in the same
bundle across all apps?
Referring to a bundle in OSGi world is like referring to a JAR in a normal
Java environment. One would not always expect to resolve class A.B.C from
bundle xyz-1.2.3.jar, right? You would resolve from the classpath.
There are other mechanisms to reach into the OSGi environment and resolve a
class arbitrarily from another bundle. Some of which I've pointed out. But
relying on a bundle symbolic name receives an -1 from me.
Can't elaborate now, I'm travelling.
Will get back to my office tomorrow and I'll dig deeper into possibilities.
Regards,
Raúl.
On 4 Nov 2015 03:07, "Dmitriy Setrakyan" <ds...@apache.org> wrote:
> Andrey, et al,
>
> Can you provide a way on how to get a required Bundle object based on, say,
> bundle symbolic name and version?
>
> I have reached the end of the internet trying to find a way in OSGI to look
> up a Bundle based on something other than an actual Class belonging to that
> bundle, but no luck.
>
> D.
>
> On Tue, Nov 3, 2015 at 5:15 PM, Andrey Kornev <an...@hotmail.com>
> wrote:
>
> > Dmitriy,
> >
> > I think your approach will work, but I let Romain respond.
> >
> > Also, in terms of the implementation, please keep in mind that the
> > resolver must be called for each non-JDK and non-Ignite core class (it
> > would probably make sense to eschew storing class loaders for such
> classes
> > in favor of compactness of the serialized representation -- see below).
> > Also, it's worth keeping a cache of already resolved class loaders per
> > marshaller invocation (this is probably the context that Romain has
> > mentioned in his previous posting) to minimize the number of the resolver
> > calls.
> >
> > In terms of the resolver's implementation, the simplest way to serialize
> > the class loader would be by capturing two pieces of information (both
> > strings): the bundle symbolic name and the bundle version. This approach
> > however may result in bloating of the serialized representation: I'd
> > roughly estimate the overhead per element to be at least 20-30 bytes (the
> > length of the symbolic name string, plus the length of the version
> string).
> > There are way to reduce the overhead (like serializing the hash code of
> the
> > bundle id string rather than the string itself, and then come up with a
> > clever way of resolving the hash collisions), but they all come at cost
> of
> > increased complexity...
> >
> > An alternative approach would be rely on the special bundle id which is
> an
> > integer and is generated by the OSGi container. But in this case, all
> nodes
> > must ensure that all the bundles have consistent ids (bundle A with id 42
> > on node N1, has the same id on every other node) which is not easy --
> while
> > not entirely impossible -- to guarantee. As long as the nodes are
> > homogeneous (have the same set of bundles deployed) the OSGi container is
> > guaranteed to assign to the bundles the same ids.
> >
> > Thanks
> > Andrey
> >
> > > From: dsetrakyan@apache.org
> > > Date: Tue, 3 Nov 2015 16:29:41 -0800
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: dev@ignite.apache.org
> > >
> > > Romain,
> > >
> > > In the upcoming release we will be deprecating the OptimizedMarshaller
> > and
> > > will be switching to a default internal marshaller (which is based on
> the
> > > new PortableMarshaller donated by GridGain).
> > >
> > > Having said that, we may be able to pass BinaryWriter and BinaryReader
> > > instead of byte arrays. This will be pretty close to passing the
> stream,
> > as
> > > suggested by Andrey.
> > >
> > > Also, I still think that we should only resolve class loaders and not
> the
> > > class itself. The main reason is that Ignite will encode class names
> into
> > > an integer hash code and will store the integer->class-fqn mapping in
> > > internal replicated cache. I doubt users will get more efficient than
> an
> > > integer (4 bytes) for a class name.
> > >
> > > On the receiving side, once we are able to get the right class loader,
> we
> > > can easily get the proper class by calling
> > ClassLoader.findClass(class-fqn).
> > >
> > > Thoughts?
> > >
> > > D.
> > >
> > > On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <romain.gilles@gmail.com
> >
> > > wrote:
> > >
> > > > Hi,
> > > > Maybe a missing point. I think but I'm not sure that in the
> > > > OptimizedMarshaller there is a caching of already serialized classes.
> > Due
> > > > to the dynamic nature of OSGi this may lead to memory leak. In fact
> if
> > a
> > > > bundle is refreshed, it will produce a new BundleRevision and
> > therefore a
> > > > new classloader. And if you don't release the class from the previous
> > > > BundleRevision then you endup with memory leak. So maybe the
> Marshaller
> > > > interface or somewhere should provide a way to free those classes /
> > > > classloaders.
> > > >
> > > > Regards,
> > > >
> > > > Romain
> > > >
> > > > Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <andrewkornev@hotmail.com
> >
> > a
> > > > écrit :
> > > >
> > > > > Romain/Dmitriy,
> > > > >
> > > > > I prefer Romain's approach, but just curious, in the API you guys
> are
> > > > > proposing why use a byte[] rather than OutputStream/InputStream?
> > With a
> > > > > byte[], one would inevitably end up wrapping it into a byte stream
> > class.
> > > > > Also, the stream-based interface would be more in line with the
> > > > Marshaller
> > > > > API.
> > > > >
> > > > > Also for symmetry with the readClass() method, I suggest the
> > writeClass()
> > > > > take a Class<?> rather than an object.
> > > > >
> > > > > Regards
> > > > > Andrey
> > > > >
> > > > > > From: romain.gilles@gmail.com
> > > > > > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > To: dev@ignite.apache.org
> > > > > >
> > > > > > Hi Dmitriy,
> > > > > > I think your solution is good. Maybe I will change it a little
> > bit...
> > > > :P
> > > > > > I think you should delegate the Class resolution to the resolver.
> > > > Because
> > > > > > for example with your current solution the marshaller may before
> > (or
> > > > > after)
> > > > > > store the fqn of the class (maybe only the first time it
> encounter
> > it)
> > > > > but
> > > > > > in order to save the classloader context resolution the
> > implementation
> > > > of
> > > > > > the resolver may have to save the package name of the given
> object
> > and
> > > > > some
> > > > > > extra info therefore the content package name will be serialized
> > two
> > > > > times.
> > > > > > Ok, it's not a big deal.
> > > > > > But now if the resolver identify that it can reuse the same class
> > > > loader
> > > > > > for a couple of classes. It will be interesting for it to have a
> > > > > > serialization context in order to save, let say, classloader/id
> > mapping
> > > > > in
> > > > > > order to save the id instead of a more longer representation.
> > > > > > So I propose something like that:
> > > > > > *public interface ClassResolver {*
> > > > > > * // This method is invoked on the sender side to *
> > > > > > * // marshal the information about the class.*
> > > > > > * // where the context is a map style object that is reset/new
> > for
> > > > > each
> > > > > > object graph serialization.*
> > > > > > * public byte[] writeClass(Object o, Context context) throws
> > > > > > IgniteException;*
> > > > > >
> > > > > > * // This method is invoked on the receiving side to*
> > > > > > * // retrieve the class based on provided information.*
> > > > > > * // where the context is a map style object that is reset/new
> > for
> > > > > each
> > > > > > object graph serialization.*
> > > > > > * public Class<?> readClass(byte[], Context context) throws
> > > > > > IgniteException;*
> > > > > > *}*
> > > > > >
> > > > > > I think your proposal will solve our issue and maybe also open a
> > door
> > > > for
> > > > > > the osgi development.
> > > > > > Let me know what do you think about me proposal? :)
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Romain
> > > > > >
> > > > > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > > > a
> > > > > > écrit :
> > > > > >
> > > > > > > Romain,
> > > > > > >
> > > > > > > Can you comment on the ClassLoaderResolver suggestion that I
> > proposed
> > > > > > > earlier? Will it solve your problem?
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <
> > > > romain.gilles@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Raul,
> > > > > > > > - Do you plan to use the TCCL when marshalling in OSGi env?
> > If yes
> > > > > how?
> > > > > > > > - I like the point 2. Maybe for a graph of object that come
> > from
> > > > > > > different
> > > > > > > > packages / bundles you may have to recursively capture the
> > package
> > > > > > > version
> > > > > > > > of the individual element of your graph.
> > > > > > > > - For the point 3 I wonder if it will not over-complexify
> the
> > > > > solution.
> > > > > > > As
> > > > > > > > a developer you will have to think about it. And it is not
> > obvious
> > > > > in the
> > > > > > > > code where things are serialized or not. You may use lambda
> in
> > your
> > > > > > > > application code therefore the current package become what
> you
> > call
> > > > > a DTO
> > > > > > > > package. The current state of ignite does not enforce you to
> > > > specify
> > > > > it
> > > > > > > for
> > > > > > > > "classical" classloading application. If you introduce this
> > extra
> > > > > step
> > > > > > > for
> > > > > > > > OSGi ready application you will maybe discourage people to
> use
> > > > OSGi.
> > > > > > > >
> > > > > > > > To comeback to our solution. We start we a strong assumption:
> > our
> > > > > cluster
> > > > > > > > is homogeneous in term of code so, of course, it simplify our
> > life
> > > > > :).
> > > > > > > >
> > > > > > > > BTW if you can introduce an extension point in the class
> > resolution
> > > > > > > > mechanism it can be interesting for end user in order to
> allow
> > them
> > > > > to
> > > > > > > > optimize it based on there specific use cases.
> > > > > > > >
> > > > > > > > Romain.
> > > > > > > >
> > > > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <
> raul@evosent.com>
> > a
> > > > > écrit :
> > > > > > > >
> > > > > > > > > Hi Andrey,
> > > > > > > > >
> > > > > > > > > Thanks for the participation in this topic.
> > > > > > > > >
> > > > > > > > > I don't like the solution to incorporate the bundle
> symbolic
> > name
> > > > > in
> > > > > > > the
> > > > > > > > > serialised form. Nothing guarantees that the classdef will
> be
> > > > > located
> > > > > > > > under
> > > > > > > > > the same bundle in both source and target machines. We also
> > have
> > > > to
> > > > > > > take
> > > > > > > > > into account that OSGi allows for multiple versions of the
> > same
> > > > > > > > > bundle/packages to co-exist in the same container. So it
> > becomes
> > > > > more
> > > > > > > > > complex.
> > > > > > > > >
> > > > > > > > > Using the TCCL should work when serialising, but it'll
> > probably
> > > > be
> > > > > of
> > > > > > > no
> > > > > > > > > use when deserialising on the other end.
> > > > > > > > >
> > > > > > > > > I need to enhance Ignite to:
> > > > > > > > >
> > > > > > > > > 1. Use the TCCL when marshalling on one end.
> > > > > > > > > 2. Incorporate the package version of the class in the
> > serialised
> > > > > form
> > > > > > > if
> > > > > > > > > Ignite is running in an OSGi environment.
> > > > > > > > > 3. On the receiver end, discover cache entities / DTOs in
> all
> > > > > bundles
> > > > > > > > > through a custom OSGi manifest header or the like, as I
> > explained
> > > > > > > before.
> > > > > > > > > Package version must be taken into account.
> > > > > > > > >
> > > > > > > > > What do you think?
> > > > > > > > >
> > > > > > > > > Raúl.
> > > > > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <
> > andrewkornev@hotmail.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Raul,
> > > > > > > > > >
> > > > > > > > > > The fundamental hurdle we need to jump over to make
> Ignite
> > > > > > > OSGi-enabled
> > > > > > > > > is
> > > > > > > > > > the marshalling. More specifically the issue is with
> > > > > deserialization
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > classes that are provided by the bundles other than the
> > Ignite
> > > > > bundle
> > > > > > > > > > itself.
> > > > > > > > > >
> > > > > > > > > > When the Ignite transport layer receives a message it
> > needs to
> > > > > figure
> > > > > > > > out
> > > > > > > > > > how to deserialize the bytes and for that it needs to
> know
> > the
> > > > > bundle
> > > > > > > > > that
> > > > > > > > > > provides the class to be deserailized. At this point TCCL
> > is of
> > > > > no
> > > > > > > use.
> > > > > > > > > To
> > > > > > > > > > make things more complex, the class may contain other
> > classes
> > > > > that
> > > > > > > come
> > > > > > > > > > from other bundles, and so on recursively. This means
> that
> > each
> > > > > > > object
> > > > > > > > in
> > > > > > > > > > the hierarchy must be serialized with its bundle name (or
> > > > bundle
> > > > > id),
> > > > > > > > so
> > > > > > > > > > that the deserializer will then be able to correctly
> > resolve
> > > > the
> > > > > > > class
> > > > > > > > > > while traversing the object hierarchy during
> > deserialization.
> > > > > > > > > >
> > > > > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking
> the
> > > > > ability to
> > > > > > > > > plug
> > > > > > > > > > in a custom class resolver. Romain's solution was to use
> > Kryo
> > > > > that
> > > > > > > does
> > > > > > > > > > provide a way to customize class resolution. It has
> solved
> > the
> > > > > > > problem
> > > > > > > > of
> > > > > > > > > > capturing the bundle info and he was able to successfully
> > run
> > > > > Ignite
> > > > > > > > as a
> > > > > > > > > > bundle in an OSGi container (after some repackaging and
> > > > > inclusion of
> > > > > > > > the
> > > > > > > > > > manifest). But Kryo-based marshalling introduced a lot of
> > > > > complexity
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > code and incorrect use of Kryo's numerous serializers
> > caused
> > > > some
> > > > > > > weird
> > > > > > > > > > hard-to-debug issues in the Ignite core (like duplicate
> > cache
> > > > > entries
> > > > > > > > due
> > > > > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap
> > class --
> > > > > go
> > > > > > > > > > figure!). Overall the Kryo-based solution is brittle and
> > hard
> > > > to
> > > > > > > > > maintain.
> > > > > > > > > >
> > > > > > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > > > > > 1) enhance the OptimizedMarshaller to allow custom class
> > > > > resolution.
> > > > > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in
> > addition to
> > > > > the
> > > > > > > > > > original one) to be used in OSGi environment.
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > > Andrey
> > > > > > > > > >
> > > > > > > > > > > From: raulk@apache.org
> > > > > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > > > > > Subject: Re: OSGi migration may require a special
> > marshaller
> > > > > > > > > > > To: dev@ignite.apache.org
> > > > > > > > > > >
> > > > > > > > > > > Hi Romain,
> > > > > > > > > > >
> > > > > > > > > > > I'm working on the OSGi compatibility of Ignite. I
> > appreciate
> > > > > your
> > > > > > > > > input.
> > > > > > > > > > >
> > > > > > > > > > > I'm thinking about the situation you describe and I
> > wonder if
> > > > > > > you're
> > > > > > > > > > > exporting Ignite as an OSGi service which is then
> > consumed
> > > > from
> > > > > > > other
> > > > > > > > > > > bundles. Under this situation, it would be quite easy
> to
> > > > > reproduce
> > > > > > > > the
> > > > > > > > > > > behaviour you describe if Ignite is not resolving
> > classes via
> > > > > the
> > > > > > > > TCCL.
> > > > > > > > > > > Need to dig deeper into that.
> > > > > > > > > > >
> > > > > > > > > > > Off the top of my head, there are two alternatives to
> > solve
> > > > it:
> > > > > > > > > > >
> > > > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not
> > already
> > > > > > > used) –
> > > > > > > > > we
> > > > > > > > > > > gotta be wary of possible regressions.
> > > > > > > > > > > 2. Create a special OSGi header Ignite-Export-Package
> so
> > that
> > > > > > > bundles
> > > > > > > > > > > containing DTOs can expose packages to Ignite's
> > marshallers.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > >
> > > > > > > > > > > *Raúl Kripalani*
> > > > > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel |
> > Integration,
> > > > > Big
> > > > > > > Data
> > > > > > > > > and
> > > > > > > > > > > Messaging Engineer
> > > > > > > > > > > http://about.me/raulkripalani |
> > > > > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > > > > > romain.gilles@misys.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm really interested in this issue:
> > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 .
> We
> > > > some
> > > > > > > stuff
> > > > > > > > to
> > > > > > > > > > make
> > > > > > > > > > > > it work in our osgi environment. The main issue for
> us
> > now
> > > > > is the
> > > > > > > > > > > > serialization. I think it you will have to rework the
> > > > > > > > > > OptimizedMarshaller
> > > > > > > > > > > > or any other marshaller that works with object that
> > come
> > > > from
> > > > > > > > outside
> > > > > > > > > > your
> > > > > > > > > > > > class space.
> > > > > > > > > > > >
> > > > > > > > > > > > We have try kryo that works. Kryo provide an
> extension
> > > > point
> > > > > in
> > > > > > > > order
> > > > > > > > > > to
> > > > > > > > > > > > resolve the classes:
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > > > > > . With this extension we are able to solve the
> problem
> > of
> > > > > > > external
> > > > > > > > > > classes.
> > > > > > > > > > > > The only issue with kryo is that some classes need a
> > > > certain
> > > > > care
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > serialization process and therefore a specialized
> > > > serializer.
> > > > > > > > > > > >
> > > > > > > > > > > > So I would like to know from the community what do
> > think of
> > > > > > > > changing
> > > > > > > > > > the
> > > > > > > > > > > > way the optimized marshaller works or introducing the
> > > > > support of
> > > > > > > > yet
> > > > > > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks in advance,
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Romain.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > > > > > "Misys" is the trade name of the Misys group of
> > companies.
> > > > > This
> > > > > > > > email
> > > > > > > > > > and
> > > > > > > > > > > > any attachments have been scanned for known viruses
> > using
> > > > > > > multiple
> > > > > > > > > > > > scanners. This email message is intended for the
> named
> > > > > recipient
> > > > > > > > > only.
> > > > > > > > > > It
> > > > > > > > > > > > may be privileged and/or confidential. If you are not
> > the
> > > > > named
> > > > > > > > > > recipient
> > > > > > > > > > > > of this email please notify us immediately and do not
> > copy
> > > > > it or
> > > > > > > > use
> > > > > > > > > > it for
> > > > > > > > > > > > any purpose, nor disclose its contents to any other
> > person.
> > > > > This
> > > > > > > > > email
> > > > > > > > > > does
> > > > > > > > > > > > not constitute the commencement of legal relations
> > between
> > > > > you
> > > > > > > and
> > > > > > > > > > Misys.
> > > > > > > > > > > > Please refer to the executed contract between you and
> > the
> > > > > > > relevant
> > > > > > > > > > member
> > > > > > > > > > > > of the Misys group for the identity of the
> contracting
> > > > party
> > > > > with
> > > > > > > > > > which you
> > > > > > > > > > > > are dealing.
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Andrey, et al,
Can you provide a way on how to get a required Bundle object based on, say,
bundle symbolic name and version?
I have reached the end of the internet trying to find a way in OSGI to look
up a Bundle based on something other than an actual Class belonging to that
bundle, but no luck.
D.
On Tue, Nov 3, 2015 at 5:15 PM, Andrey Kornev <an...@hotmail.com>
wrote:
> Dmitriy,
>
> I think your approach will work, but I let Romain respond.
>
> Also, in terms of the implementation, please keep in mind that the
> resolver must be called for each non-JDK and non-Ignite core class (it
> would probably make sense to eschew storing class loaders for such classes
> in favor of compactness of the serialized representation -- see below).
> Also, it's worth keeping a cache of already resolved class loaders per
> marshaller invocation (this is probably the context that Romain has
> mentioned in his previous posting) to minimize the number of the resolver
> calls.
>
> In terms of the resolver's implementation, the simplest way to serialize
> the class loader would be by capturing two pieces of information (both
> strings): the bundle symbolic name and the bundle version. This approach
> however may result in bloating of the serialized representation: I'd
> roughly estimate the overhead per element to be at least 20-30 bytes (the
> length of the symbolic name string, plus the length of the version string).
> There are way to reduce the overhead (like serializing the hash code of the
> bundle id string rather than the string itself, and then come up with a
> clever way of resolving the hash collisions), but they all come at cost of
> increased complexity...
>
> An alternative approach would be rely on the special bundle id which is an
> integer and is generated by the OSGi container. But in this case, all nodes
> must ensure that all the bundles have consistent ids (bundle A with id 42
> on node N1, has the same id on every other node) which is not easy -- while
> not entirely impossible -- to guarantee. As long as the nodes are
> homogeneous (have the same set of bundles deployed) the OSGi container is
> guaranteed to assign to the bundles the same ids.
>
> Thanks
> Andrey
>
> > From: dsetrakyan@apache.org
> > Date: Tue, 3 Nov 2015 16:29:41 -0800
> > Subject: Re: OSGi migration may require a special marshaller
> > To: dev@ignite.apache.org
> >
> > Romain,
> >
> > In the upcoming release we will be deprecating the OptimizedMarshaller
> and
> > will be switching to a default internal marshaller (which is based on the
> > new PortableMarshaller donated by GridGain).
> >
> > Having said that, we may be able to pass BinaryWriter and BinaryReader
> > instead of byte arrays. This will be pretty close to passing the stream,
> as
> > suggested by Andrey.
> >
> > Also, I still think that we should only resolve class loaders and not the
> > class itself. The main reason is that Ignite will encode class names into
> > an integer hash code and will store the integer->class-fqn mapping in
> > internal replicated cache. I doubt users will get more efficient than an
> > integer (4 bytes) for a class name.
> >
> > On the receiving side, once we are able to get the right class loader, we
> > can easily get the proper class by calling
> ClassLoader.findClass(class-fqn).
> >
> > Thoughts?
> >
> > D.
> >
> > On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <ro...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > Maybe a missing point. I think but I'm not sure that in the
> > > OptimizedMarshaller there is a caching of already serialized classes.
> Due
> > > to the dynamic nature of OSGi this may lead to memory leak. In fact if
> a
> > > bundle is refreshed, it will produce a new BundleRevision and
> therefore a
> > > new classloader. And if you don't release the class from the previous
> > > BundleRevision then you endup with memory leak. So maybe the Marshaller
> > > interface or somewhere should provide a way to free those classes /
> > > classloaders.
> > >
> > > Regards,
> > >
> > > Romain
> > >
> > > Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <an...@hotmail.com>
> a
> > > écrit :
> > >
> > > > Romain/Dmitriy,
> > > >
> > > > I prefer Romain's approach, but just curious, in the API you guys are
> > > > proposing why use a byte[] rather than OutputStream/InputStream?
> With a
> > > > byte[], one would inevitably end up wrapping it into a byte stream
> class.
> > > > Also, the stream-based interface would be more in line with the
> > > Marshaller
> > > > API.
> > > >
> > > > Also for symmetry with the readClass() method, I suggest the
> writeClass()
> > > > take a Class<?> rather than an object.
> > > >
> > > > Regards
> > > > Andrey
> > > >
> > > > > From: romain.gilles@gmail.com
> > > > > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > To: dev@ignite.apache.org
> > > > >
> > > > > Hi Dmitriy,
> > > > > I think your solution is good. Maybe I will change it a little
> bit...
> > > :P
> > > > > I think you should delegate the Class resolution to the resolver.
> > > Because
> > > > > for example with your current solution the marshaller may before
> (or
> > > > after)
> > > > > store the fqn of the class (maybe only the first time it encounter
> it)
> > > > but
> > > > > in order to save the classloader context resolution the
> implementation
> > > of
> > > > > the resolver may have to save the package name of the given object
> and
> > > > some
> > > > > extra info therefore the content package name will be serialized
> two
> > > > times.
> > > > > Ok, it's not a big deal.
> > > > > But now if the resolver identify that it can reuse the same class
> > > loader
> > > > > for a couple of classes. It will be interesting for it to have a
> > > > > serialization context in order to save, let say, classloader/id
> mapping
> > > > in
> > > > > order to save the id instead of a more longer representation.
> > > > > So I propose something like that:
> > > > > *public interface ClassResolver {*
> > > > > * // This method is invoked on the sender side to *
> > > > > * // marshal the information about the class.*
> > > > > * // where the context is a map style object that is reset/new
> for
> > > > each
> > > > > object graph serialization.*
> > > > > * public byte[] writeClass(Object o, Context context) throws
> > > > > IgniteException;*
> > > > >
> > > > > * // This method is invoked on the receiving side to*
> > > > > * // retrieve the class based on provided information.*
> > > > > * // where the context is a map style object that is reset/new
> for
> > > > each
> > > > > object graph serialization.*
> > > > > * public Class<?> readClass(byte[], Context context) throws
> > > > > IgniteException;*
> > > > > *}*
> > > > >
> > > > > I think your proposal will solve our issue and maybe also open a
> door
> > > for
> > > > > the osgi development.
> > > > > Let me know what do you think about me proposal? :)
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Romain
> > > > >
> > > > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > > a
> > > > > écrit :
> > > > >
> > > > > > Romain,
> > > > > >
> > > > > > Can you comment on the ClassLoaderResolver suggestion that I
> proposed
> > > > > > earlier? Will it solve your problem?
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <
> > > romain.gilles@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Raul,
> > > > > > > - Do you plan to use the TCCL when marshalling in OSGi env?
> If yes
> > > > how?
> > > > > > > - I like the point 2. Maybe for a graph of object that come
> from
> > > > > > different
> > > > > > > packages / bundles you may have to recursively capture the
> package
> > > > > > version
> > > > > > > of the individual element of your graph.
> > > > > > > - For the point 3 I wonder if it will not over-complexify the
> > > > solution.
> > > > > > As
> > > > > > > a developer you will have to think about it. And it is not
> obvious
> > > > in the
> > > > > > > code where things are serialized or not. You may use lambda in
> your
> > > > > > > application code therefore the current package become what you
> call
> > > > a DTO
> > > > > > > package. The current state of ignite does not enforce you to
> > > specify
> > > > it
> > > > > > for
> > > > > > > "classical" classloading application. If you introduce this
> extra
> > > > step
> > > > > > for
> > > > > > > OSGi ready application you will maybe discourage people to use
> > > OSGi.
> > > > > > >
> > > > > > > To comeback to our solution. We start we a strong assumption:
> our
> > > > cluster
> > > > > > > is homogeneous in term of code so, of course, it simplify our
> life
> > > > :).
> > > > > > >
> > > > > > > BTW if you can introduce an extension point in the class
> resolution
> > > > > > > mechanism it can be interesting for end user in order to allow
> them
> > > > to
> > > > > > > optimize it based on there specific use cases.
> > > > > > >
> > > > > > > Romain.
> > > > > > >
> > > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <ra...@evosent.com>
> a
> > > > écrit :
> > > > > > >
> > > > > > > > Hi Andrey,
> > > > > > > >
> > > > > > > > Thanks for the participation in this topic.
> > > > > > > >
> > > > > > > > I don't like the solution to incorporate the bundle symbolic
> name
> > > > in
> > > > > > the
> > > > > > > > serialised form. Nothing guarantees that the classdef will be
> > > > located
> > > > > > > under
> > > > > > > > the same bundle in both source and target machines. We also
> have
> > > to
> > > > > > take
> > > > > > > > into account that OSGi allows for multiple versions of the
> same
> > > > > > > > bundle/packages to co-exist in the same container. So it
> becomes
> > > > more
> > > > > > > > complex.
> > > > > > > >
> > > > > > > > Using the TCCL should work when serialising, but it'll
> probably
> > > be
> > > > of
> > > > > > no
> > > > > > > > use when deserialising on the other end.
> > > > > > > >
> > > > > > > > I need to enhance Ignite to:
> > > > > > > >
> > > > > > > > 1. Use the TCCL when marshalling on one end.
> > > > > > > > 2. Incorporate the package version of the class in the
> serialised
> > > > form
> > > > > > if
> > > > > > > > Ignite is running in an OSGi environment.
> > > > > > > > 3. On the receiver end, discover cache entities / DTOs in all
> > > > bundles
> > > > > > > > through a custom OSGi manifest header or the like, as I
> explained
> > > > > > before.
> > > > > > > > Package version must be taken into account.
> > > > > > > >
> > > > > > > > What do you think?
> > > > > > > >
> > > > > > > > Raúl.
> > > > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <
> andrewkornev@hotmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Raul,
> > > > > > > > >
> > > > > > > > > The fundamental hurdle we need to jump over to make Ignite
> > > > > > OSGi-enabled
> > > > > > > > is
> > > > > > > > > the marshalling. More specifically the issue is with
> > > > deserialization
> > > > > > of
> > > > > > > > the
> > > > > > > > > classes that are provided by the bundles other than the
> Ignite
> > > > bundle
> > > > > > > > > itself.
> > > > > > > > >
> > > > > > > > > When the Ignite transport layer receives a message it
> needs to
> > > > figure
> > > > > > > out
> > > > > > > > > how to deserialize the bytes and for that it needs to know
> the
> > > > bundle
> > > > > > > > that
> > > > > > > > > provides the class to be deserailized. At this point TCCL
> is of
> > > > no
> > > > > > use.
> > > > > > > > To
> > > > > > > > > make things more complex, the class may contain other
> classes
> > > > that
> > > > > > come
> > > > > > > > > from other bundles, and so on recursively. This means that
> each
> > > > > > object
> > > > > > > in
> > > > > > > > > the hierarchy must be serialized with its bundle name (or
> > > bundle
> > > > id),
> > > > > > > so
> > > > > > > > > that the deserializer will then be able to correctly
> resolve
> > > the
> > > > > > class
> > > > > > > > > while traversing the object hierarchy during
> deserialization.
> > > > > > > > >
> > > > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the
> > > > ability to
> > > > > > > > plug
> > > > > > > > > in a custom class resolver. Romain's solution was to use
> Kryo
> > > > that
> > > > > > does
> > > > > > > > > provide a way to customize class resolution. It has solved
> the
> > > > > > problem
> > > > > > > of
> > > > > > > > > capturing the bundle info and he was able to successfully
> run
> > > > Ignite
> > > > > > > as a
> > > > > > > > > bundle in an OSGi container (after some repackaging and
> > > > inclusion of
> > > > > > > the
> > > > > > > > > manifest). But Kryo-based marshalling introduced a lot of
> > > > complexity
> > > > > > to
> > > > > > > > the
> > > > > > > > > code and incorrect use of Kryo's numerous serializers
> caused
> > > some
> > > > > > weird
> > > > > > > > > hard-to-debug issues in the Ignite core (like duplicate
> cache
> > > > entries
> > > > > > > due
> > > > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap
> class --
> > > > go
> > > > > > > > > figure!). Overall the Kryo-based solution is brittle and
> hard
> > > to
> > > > > > > > maintain.
> > > > > > > > >
> > > > > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > > > > 1) enhance the OptimizedMarshaller to allow custom class
> > > > resolution.
> > > > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in
> addition to
> > > > the
> > > > > > > > > original one) to be used in OSGi environment.
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > Andrey
> > > > > > > > >
> > > > > > > > > > From: raulk@apache.org
> > > > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > > > > Subject: Re: OSGi migration may require a special
> marshaller
> > > > > > > > > > To: dev@ignite.apache.org
> > > > > > > > > >
> > > > > > > > > > Hi Romain,
> > > > > > > > > >
> > > > > > > > > > I'm working on the OSGi compatibility of Ignite. I
> appreciate
> > > > your
> > > > > > > > input.
> > > > > > > > > >
> > > > > > > > > > I'm thinking about the situation you describe and I
> wonder if
> > > > > > you're
> > > > > > > > > > exporting Ignite as an OSGi service which is then
> consumed
> > > from
> > > > > > other
> > > > > > > > > > bundles. Under this situation, it would be quite easy to
> > > > reproduce
> > > > > > > the
> > > > > > > > > > behaviour you describe if Ignite is not resolving
> classes via
> > > > the
> > > > > > > TCCL.
> > > > > > > > > > Need to dig deeper into that.
> > > > > > > > > >
> > > > > > > > > > Off the top of my head, there are two alternatives to
> solve
> > > it:
> > > > > > > > > >
> > > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not
> already
> > > > > > used) –
> > > > > > > > we
> > > > > > > > > > gotta be wary of possible regressions.
> > > > > > > > > > 2. Create a special OSGi header Ignite-Export-Package so
> that
> > > > > > bundles
> > > > > > > > > > containing DTOs can expose packages to Ignite's
> marshallers.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > >
> > > > > > > > > > *Raúl Kripalani*
> > > > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel |
> Integration,
> > > > Big
> > > > > > Data
> > > > > > > > and
> > > > > > > > > > Messaging Engineer
> > > > > > > > > > http://about.me/raulkripalani |
> > > > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > > > > >
> > > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > > > > romain.gilles@misys.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I'm really interested in this issue:
> > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We
> > > some
> > > > > > stuff
> > > > > > > to
> > > > > > > > > make
> > > > > > > > > > > it work in our osgi environment. The main issue for us
> now
> > > > is the
> > > > > > > > > > > serialization. I think it you will have to rework the
> > > > > > > > > OptimizedMarshaller
> > > > > > > > > > > or any other marshaller that works with object that
> come
> > > from
> > > > > > > outside
> > > > > > > > > your
> > > > > > > > > > > class space.
> > > > > > > > > > >
> > > > > > > > > > > We have try kryo that works. Kryo provide an extension
> > > point
> > > > in
> > > > > > > order
> > > > > > > > > to
> > > > > > > > > > > resolve the classes:
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > > > > . With this extension we are able to solve the problem
> of
> > > > > > external
> > > > > > > > > classes.
> > > > > > > > > > > The only issue with kryo is that some classes need a
> > > certain
> > > > care
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > serialization process and therefore a specialized
> > > serializer.
> > > > > > > > > > >
> > > > > > > > > > > So I would like to know from the community what do
> think of
> > > > > > > changing
> > > > > > > > > the
> > > > > > > > > > > way the optimized marshaller works or introducing the
> > > > support of
> > > > > > > yet
> > > > > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks in advance,
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Romain.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > > > > "Misys" is the trade name of the Misys group of
> companies.
> > > > This
> > > > > > > email
> > > > > > > > > and
> > > > > > > > > > > any attachments have been scanned for known viruses
> using
> > > > > > multiple
> > > > > > > > > > > scanners. This email message is intended for the named
> > > > recipient
> > > > > > > > only.
> > > > > > > > > It
> > > > > > > > > > > may be privileged and/or confidential. If you are not
> the
> > > > named
> > > > > > > > > recipient
> > > > > > > > > > > of this email please notify us immediately and do not
> copy
> > > > it or
> > > > > > > use
> > > > > > > > > it for
> > > > > > > > > > > any purpose, nor disclose its contents to any other
> person.
> > > > This
> > > > > > > > email
> > > > > > > > > does
> > > > > > > > > > > not constitute the commencement of legal relations
> between
> > > > you
> > > > > > and
> > > > > > > > > Misys.
> > > > > > > > > > > Please refer to the executed contract between you and
> the
> > > > > > relevant
> > > > > > > > > member
> > > > > > > > > > > of the Misys group for the identity of the contracting
> > > party
> > > > with
> > > > > > > > > which you
> > > > > > > > > > > are dealing.
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
>
>
RE: OSGi migration may require a special marshaller
Posted by Andrey Kornev <an...@hotmail.com>.
Dmitriy,
I think your approach will work, but I let Romain respond.
Also, in terms of the implementation, please keep in mind that the resolver must be called for each non-JDK and non-Ignite core class (it would probably make sense to eschew storing class loaders for such classes in favor of compactness of the serialized representation -- see below). Also, it's worth keeping a cache of already resolved class loaders per marshaller invocation (this is probably the context that Romain has mentioned in his previous posting) to minimize the number of the resolver calls.
In terms of the resolver's implementation, the simplest way to serialize the class loader would be by capturing two pieces of information (both strings): the bundle symbolic name and the bundle version. This approach however may result in bloating of the serialized representation: I'd roughly estimate the overhead per element to be at least 20-30 bytes (the length of the symbolic name string, plus the length of the version string). There are way to reduce the overhead (like serializing the hash code of the bundle id string rather than the string itself, and then come up with a clever way of resolving the hash collisions), but they all come at cost of increased complexity...
An alternative approach would be rely on the special bundle id which is an integer and is generated by the OSGi container. But in this case, all nodes must ensure that all the bundles have consistent ids (bundle A with id 42 on node N1, has the same id on every other node) which is not easy -- while not entirely impossible -- to guarantee. As long as the nodes are homogeneous (have the same set of bundles deployed) the OSGi container is guaranteed to assign to the bundles the same ids.
Thanks
Andrey
> From: dsetrakyan@apache.org
> Date: Tue, 3 Nov 2015 16:29:41 -0800
> Subject: Re: OSGi migration may require a special marshaller
> To: dev@ignite.apache.org
>
> Romain,
>
> In the upcoming release we will be deprecating the OptimizedMarshaller and
> will be switching to a default internal marshaller (which is based on the
> new PortableMarshaller donated by GridGain).
>
> Having said that, we may be able to pass BinaryWriter and BinaryReader
> instead of byte arrays. This will be pretty close to passing the stream, as
> suggested by Andrey.
>
> Also, I still think that we should only resolve class loaders and not the
> class itself. The main reason is that Ignite will encode class names into
> an integer hash code and will store the integer->class-fqn mapping in
> internal replicated cache. I doubt users will get more efficient than an
> integer (4 bytes) for a class name.
>
> On the receiving side, once we are able to get the right class loader, we
> can easily get the proper class by calling ClassLoader.findClass(class-fqn).
>
> Thoughts?
>
> D.
>
> On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <ro...@gmail.com>
> wrote:
>
> > Hi,
> > Maybe a missing point. I think but I'm not sure that in the
> > OptimizedMarshaller there is a caching of already serialized classes. Due
> > to the dynamic nature of OSGi this may lead to memory leak. In fact if a
> > bundle is refreshed, it will produce a new BundleRevision and therefore a
> > new classloader. And if you don't release the class from the previous
> > BundleRevision then you endup with memory leak. So maybe the Marshaller
> > interface or somewhere should provide a way to free those classes /
> > classloaders.
> >
> > Regards,
> >
> > Romain
> >
> > Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <an...@hotmail.com> a
> > écrit :
> >
> > > Romain/Dmitriy,
> > >
> > > I prefer Romain's approach, but just curious, in the API you guys are
> > > proposing why use a byte[] rather than OutputStream/InputStream? With a
> > > byte[], one would inevitably end up wrapping it into a byte stream class.
> > > Also, the stream-based interface would be more in line with the
> > Marshaller
> > > API.
> > >
> > > Also for symmetry with the readClass() method, I suggest the writeClass()
> > > take a Class<?> rather than an object.
> > >
> > > Regards
> > > Andrey
> > >
> > > > From: romain.gilles@gmail.com
> > > > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > > > Subject: Re: OSGi migration may require a special marshaller
> > > > To: dev@ignite.apache.org
> > > >
> > > > Hi Dmitriy,
> > > > I think your solution is good. Maybe I will change it a little bit...
> > :P
> > > > I think you should delegate the Class resolution to the resolver.
> > Because
> > > > for example with your current solution the marshaller may before (or
> > > after)
> > > > store the fqn of the class (maybe only the first time it encounter it)
> > > but
> > > > in order to save the classloader context resolution the implementation
> > of
> > > > the resolver may have to save the package name of the given object and
> > > some
> > > > extra info therefore the content package name will be serialized two
> > > times.
> > > > Ok, it's not a big deal.
> > > > But now if the resolver identify that it can reuse the same class
> > loader
> > > > for a couple of classes. It will be interesting for it to have a
> > > > serialization context in order to save, let say, classloader/id mapping
> > > in
> > > > order to save the id instead of a more longer representation.
> > > > So I propose something like that:
> > > > *public interface ClassResolver {*
> > > > * // This method is invoked on the sender side to *
> > > > * // marshal the information about the class.*
> > > > * // where the context is a map style object that is reset/new for
> > > each
> > > > object graph serialization.*
> > > > * public byte[] writeClass(Object o, Context context) throws
> > > > IgniteException;*
> > > >
> > > > * // This method is invoked on the receiving side to*
> > > > * // retrieve the class based on provided information.*
> > > > * // where the context is a map style object that is reset/new for
> > > each
> > > > object graph serialization.*
> > > > * public Class<?> readClass(byte[], Context context) throws
> > > > IgniteException;*
> > > > *}*
> > > >
> > > > I think your proposal will solve our issue and maybe also open a door
> > for
> > > > the osgi development.
> > > > Let me know what do you think about me proposal? :)
> > > >
> > > > Thanks,
> > > >
> > > > Romain
> > > >
> > > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <ds...@apache.org>
> > a
> > > > écrit :
> > > >
> > > > > Romain,
> > > > >
> > > > > Can you comment on the ClassLoaderResolver suggestion that I proposed
> > > > > earlier? Will it solve your problem?
> > > > >
> > > > > D.
> > > > >
> > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <
> > romain.gilles@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Raul,
> > > > > > - Do you plan to use the TCCL when marshalling in OSGi env? If yes
> > > how?
> > > > > > - I like the point 2. Maybe for a graph of object that come from
> > > > > different
> > > > > > packages / bundles you may have to recursively capture the package
> > > > > version
> > > > > > of the individual element of your graph.
> > > > > > - For the point 3 I wonder if it will not over-complexify the
> > > solution.
> > > > > As
> > > > > > a developer you will have to think about it. And it is not obvious
> > > in the
> > > > > > code where things are serialized or not. You may use lambda in your
> > > > > > application code therefore the current package become what you call
> > > a DTO
> > > > > > package. The current state of ignite does not enforce you to
> > specify
> > > it
> > > > > for
> > > > > > "classical" classloading application. If you introduce this extra
> > > step
> > > > > for
> > > > > > OSGi ready application you will maybe discourage people to use
> > OSGi.
> > > > > >
> > > > > > To comeback to our solution. We start we a strong assumption: our
> > > cluster
> > > > > > is homogeneous in term of code so, of course, it simplify our life
> > > :).
> > > > > >
> > > > > > BTW if you can introduce an extension point in the class resolution
> > > > > > mechanism it can be interesting for end user in order to allow them
> > > to
> > > > > > optimize it based on there specific use cases.
> > > > > >
> > > > > > Romain.
> > > > > >
> > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <ra...@evosent.com> a
> > > écrit :
> > > > > >
> > > > > > > Hi Andrey,
> > > > > > >
> > > > > > > Thanks for the participation in this topic.
> > > > > > >
> > > > > > > I don't like the solution to incorporate the bundle symbolic name
> > > in
> > > > > the
> > > > > > > serialised form. Nothing guarantees that the classdef will be
> > > located
> > > > > > under
> > > > > > > the same bundle in both source and target machines. We also have
> > to
> > > > > take
> > > > > > > into account that OSGi allows for multiple versions of the same
> > > > > > > bundle/packages to co-exist in the same container. So it becomes
> > > more
> > > > > > > complex.
> > > > > > >
> > > > > > > Using the TCCL should work when serialising, but it'll probably
> > be
> > > of
> > > > > no
> > > > > > > use when deserialising on the other end.
> > > > > > >
> > > > > > > I need to enhance Ignite to:
> > > > > > >
> > > > > > > 1. Use the TCCL when marshalling on one end.
> > > > > > > 2. Incorporate the package version of the class in the serialised
> > > form
> > > > > if
> > > > > > > Ignite is running in an OSGi environment.
> > > > > > > 3. On the receiver end, discover cache entities / DTOs in all
> > > bundles
> > > > > > > through a custom OSGi manifest header or the like, as I explained
> > > > > before.
> > > > > > > Package version must be taken into account.
> > > > > > >
> > > > > > > What do you think?
> > > > > > >
> > > > > > > Raúl.
> > > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <an...@hotmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Raul,
> > > > > > > >
> > > > > > > > The fundamental hurdle we need to jump over to make Ignite
> > > > > OSGi-enabled
> > > > > > > is
> > > > > > > > the marshalling. More specifically the issue is with
> > > deserialization
> > > > > of
> > > > > > > the
> > > > > > > > classes that are provided by the bundles other than the Ignite
> > > bundle
> > > > > > > > itself.
> > > > > > > >
> > > > > > > > When the Ignite transport layer receives a message it needs to
> > > figure
> > > > > > out
> > > > > > > > how to deserialize the bytes and for that it needs to know the
> > > bundle
> > > > > > > that
> > > > > > > > provides the class to be deserailized. At this point TCCL is of
> > > no
> > > > > use.
> > > > > > > To
> > > > > > > > make things more complex, the class may contain other classes
> > > that
> > > > > come
> > > > > > > > from other bundles, and so on recursively. This means that each
> > > > > object
> > > > > > in
> > > > > > > > the hierarchy must be serialized with its bundle name (or
> > bundle
> > > id),
> > > > > > so
> > > > > > > > that the deserializer will then be able to correctly resolve
> > the
> > > > > class
> > > > > > > > while traversing the object hierarchy during deserialization.
> > > > > > > >
> > > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the
> > > ability to
> > > > > > > plug
> > > > > > > > in a custom class resolver. Romain's solution was to use Kryo
> > > that
> > > > > does
> > > > > > > > provide a way to customize class resolution. It has solved the
> > > > > problem
> > > > > > of
> > > > > > > > capturing the bundle info and he was able to successfully run
> > > Ignite
> > > > > > as a
> > > > > > > > bundle in an OSGi container (after some repackaging and
> > > inclusion of
> > > > > > the
> > > > > > > > manifest). But Kryo-based marshalling introduced a lot of
> > > complexity
> > > > > to
> > > > > > > the
> > > > > > > > code and incorrect use of Kryo's numerous serializers caused
> > some
> > > > > weird
> > > > > > > > hard-to-debug issues in the Ignite core (like duplicate cache
> > > entries
> > > > > > due
> > > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap class --
> > > go
> > > > > > > > figure!). Overall the Kryo-based solution is brittle and hard
> > to
> > > > > > > maintain.
> > > > > > > >
> > > > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > > > 1) enhance the OptimizedMarshaller to allow custom class
> > > resolution.
> > > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to
> > > the
> > > > > > > > original one) to be used in OSGi environment.
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Andrey
> > > > > > > >
> > > > > > > > > From: raulk@apache.org
> > > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > > > > To: dev@ignite.apache.org
> > > > > > > > >
> > > > > > > > > Hi Romain,
> > > > > > > > >
> > > > > > > > > I'm working on the OSGi compatibility of Ignite. I appreciate
> > > your
> > > > > > > input.
> > > > > > > > >
> > > > > > > > > I'm thinking about the situation you describe and I wonder if
> > > > > you're
> > > > > > > > > exporting Ignite as an OSGi service which is then consumed
> > from
> > > > > other
> > > > > > > > > bundles. Under this situation, it would be quite easy to
> > > reproduce
> > > > > > the
> > > > > > > > > behaviour you describe if Ignite is not resolving classes via
> > > the
> > > > > > TCCL.
> > > > > > > > > Need to dig deeper into that.
> > > > > > > > >
> > > > > > > > > Off the top of my head, there are two alternatives to solve
> > it:
> > > > > > > > >
> > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already
> > > > > used) –
> > > > > > > we
> > > > > > > > > gotta be wary of possible regressions.
> > > > > > > > > 2. Create a special OSGi header Ignite-Export-Package so that
> > > > > bundles
> > > > > > > > > containing DTOs can expose packages to Ignite's marshallers.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > >
> > > > > > > > > *Raúl Kripalani*
> > > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration,
> > > Big
> > > > > Data
> > > > > > > and
> > > > > > > > > Messaging Engineer
> > > > > > > > > http://about.me/raulkripalani |
> > > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > > > >
> > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > > > romain.gilles@misys.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I'm really interested in this issue:
> > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We
> > some
> > > > > stuff
> > > > > > to
> > > > > > > > make
> > > > > > > > > > it work in our osgi environment. The main issue for us now
> > > is the
> > > > > > > > > > serialization. I think it you will have to rework the
> > > > > > > > OptimizedMarshaller
> > > > > > > > > > or any other marshaller that works with object that come
> > from
> > > > > > outside
> > > > > > > > your
> > > > > > > > > > class space.
> > > > > > > > > >
> > > > > > > > > > We have try kryo that works. Kryo provide an extension
> > point
> > > in
> > > > > > order
> > > > > > > > to
> > > > > > > > > > resolve the classes:
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > > > . With this extension we are able to solve the problem of
> > > > > external
> > > > > > > > classes.
> > > > > > > > > > The only issue with kryo is that some classes need a
> > certain
> > > care
> > > > > > in
> > > > > > > > the
> > > > > > > > > > serialization process and therefore a specialized
> > serializer.
> > > > > > > > > >
> > > > > > > > > > So I would like to know from the community what do think of
> > > > > > changing
> > > > > > > > the
> > > > > > > > > > way the optimized marshaller works or introducing the
> > > support of
> > > > > > yet
> > > > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks in advance,
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Romain.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > > > "Misys" is the trade name of the Misys group of companies.
> > > This
> > > > > > email
> > > > > > > > and
> > > > > > > > > > any attachments have been scanned for known viruses using
> > > > > multiple
> > > > > > > > > > scanners. This email message is intended for the named
> > > recipient
> > > > > > > only.
> > > > > > > > It
> > > > > > > > > > may be privileged and/or confidential. If you are not the
> > > named
> > > > > > > > recipient
> > > > > > > > > > of this email please notify us immediately and do not copy
> > > it or
> > > > > > use
> > > > > > > > it for
> > > > > > > > > > any purpose, nor disclose its contents to any other person.
> > > This
> > > > > > > email
> > > > > > > > does
> > > > > > > > > > not constitute the commencement of legal relations between
> > > you
> > > > > and
> > > > > > > > Misys.
> > > > > > > > > > Please refer to the executed contract between you and the
> > > > > relevant
> > > > > > > > member
> > > > > > > > > > of the Misys group for the identity of the contracting
> > party
> > > with
> > > > > > > > which you
> > > > > > > > > > are dealing.
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
Re: OSGi migration may require a special marshaller
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Romain,
In the upcoming release we will be deprecating the OptimizedMarshaller and
will be switching to a default internal marshaller (which is based on the
new PortableMarshaller donated by GridGain).
Having said that, we may be able to pass BinaryWriter and BinaryReader
instead of byte arrays. This will be pretty close to passing the stream, as
suggested by Andrey.
Also, I still think that we should only resolve class loaders and not the
class itself. The main reason is that Ignite will encode class names into
an integer hash code and will store the integer->class-fqn mapping in
internal replicated cache. I doubt users will get more efficient than an
integer (4 bytes) for a class name.
On the receiving side, once we are able to get the right class loader, we
can easily get the proper class by calling ClassLoader.findClass(class-fqn).
Thoughts?
D.
On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <ro...@gmail.com>
wrote:
> Hi,
> Maybe a missing point. I think but I'm not sure that in the
> OptimizedMarshaller there is a caching of already serialized classes. Due
> to the dynamic nature of OSGi this may lead to memory leak. In fact if a
> bundle is refreshed, it will produce a new BundleRevision and therefore a
> new classloader. And if you don't release the class from the previous
> BundleRevision then you endup with memory leak. So maybe the Marshaller
> interface or somewhere should provide a way to free those classes /
> classloaders.
>
> Regards,
>
> Romain
>
> Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <an...@hotmail.com> a
> écrit :
>
> > Romain/Dmitriy,
> >
> > I prefer Romain's approach, but just curious, in the API you guys are
> > proposing why use a byte[] rather than OutputStream/InputStream? With a
> > byte[], one would inevitably end up wrapping it into a byte stream class.
> > Also, the stream-based interface would be more in line with the
> Marshaller
> > API.
> >
> > Also for symmetry with the readClass() method, I suggest the writeClass()
> > take a Class<?> rather than an object.
> >
> > Regards
> > Andrey
> >
> > > From: romain.gilles@gmail.com
> > > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: dev@ignite.apache.org
> > >
> > > Hi Dmitriy,
> > > I think your solution is good. Maybe I will change it a little bit...
> :P
> > > I think you should delegate the Class resolution to the resolver.
> Because
> > > for example with your current solution the marshaller may before (or
> > after)
> > > store the fqn of the class (maybe only the first time it encounter it)
> > but
> > > in order to save the classloader context resolution the implementation
> of
> > > the resolver may have to save the package name of the given object and
> > some
> > > extra info therefore the content package name will be serialized two
> > times.
> > > Ok, it's not a big deal.
> > > But now if the resolver identify that it can reuse the same class
> loader
> > > for a couple of classes. It will be interesting for it to have a
> > > serialization context in order to save, let say, classloader/id mapping
> > in
> > > order to save the id instead of a more longer representation.
> > > So I propose something like that:
> > > *public interface ClassResolver {*
> > > * // This method is invoked on the sender side to *
> > > * // marshal the information about the class.*
> > > * // where the context is a map style object that is reset/new for
> > each
> > > object graph serialization.*
> > > * public byte[] writeClass(Object o, Context context) throws
> > > IgniteException;*
> > >
> > > * // This method is invoked on the receiving side to*
> > > * // retrieve the class based on provided information.*
> > > * // where the context is a map style object that is reset/new for
> > each
> > > object graph serialization.*
> > > * public Class<?> readClass(byte[], Context context) throws
> > > IgniteException;*
> > > *}*
> > >
> > > I think your proposal will solve our issue and maybe also open a door
> for
> > > the osgi development.
> > > Let me know what do you think about me proposal? :)
> > >
> > > Thanks,
> > >
> > > Romain
> > >
> > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <ds...@apache.org>
> a
> > > écrit :
> > >
> > > > Romain,
> > > >
> > > > Can you comment on the ClassLoaderResolver suggestion that I proposed
> > > > earlier? Will it solve your problem?
> > > >
> > > > D.
> > > >
> > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <
> romain.gilles@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Raul,
> > > > > - Do you plan to use the TCCL when marshalling in OSGi env? If yes
> > how?
> > > > > - I like the point 2. Maybe for a graph of object that come from
> > > > different
> > > > > packages / bundles you may have to recursively capture the package
> > > > version
> > > > > of the individual element of your graph.
> > > > > - For the point 3 I wonder if it will not over-complexify the
> > solution.
> > > > As
> > > > > a developer you will have to think about it. And it is not obvious
> > in the
> > > > > code where things are serialized or not. You may use lambda in your
> > > > > application code therefore the current package become what you call
> > a DTO
> > > > > package. The current state of ignite does not enforce you to
> specify
> > it
> > > > for
> > > > > "classical" classloading application. If you introduce this extra
> > step
> > > > for
> > > > > OSGi ready application you will maybe discourage people to use
> OSGi.
> > > > >
> > > > > To comeback to our solution. We start we a strong assumption: our
> > cluster
> > > > > is homogeneous in term of code so, of course, it simplify our life
> > :).
> > > > >
> > > > > BTW if you can introduce an extension point in the class resolution
> > > > > mechanism it can be interesting for end user in order to allow them
> > to
> > > > > optimize it based on there specific use cases.
> > > > >
> > > > > Romain.
> > > > >
> > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <ra...@evosent.com> a
> > écrit :
> > > > >
> > > > > > Hi Andrey,
> > > > > >
> > > > > > Thanks for the participation in this topic.
> > > > > >
> > > > > > I don't like the solution to incorporate the bundle symbolic name
> > in
> > > > the
> > > > > > serialised form. Nothing guarantees that the classdef will be
> > located
> > > > > under
> > > > > > the same bundle in both source and target machines. We also have
> to
> > > > take
> > > > > > into account that OSGi allows for multiple versions of the same
> > > > > > bundle/packages to co-exist in the same container. So it becomes
> > more
> > > > > > complex.
> > > > > >
> > > > > > Using the TCCL should work when serialising, but it'll probably
> be
> > of
> > > > no
> > > > > > use when deserialising on the other end.
> > > > > >
> > > > > > I need to enhance Ignite to:
> > > > > >
> > > > > > 1. Use the TCCL when marshalling on one end.
> > > > > > 2. Incorporate the package version of the class in the serialised
> > form
> > > > if
> > > > > > Ignite is running in an OSGi environment.
> > > > > > 3. On the receiver end, discover cache entities / DTOs in all
> > bundles
> > > > > > through a custom OSGi manifest header or the like, as I explained
> > > > before.
> > > > > > Package version must be taken into account.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > Raúl.
> > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <an...@hotmail.com>
> > wrote:
> > > > > >
> > > > > > > Raul,
> > > > > > >
> > > > > > > The fundamental hurdle we need to jump over to make Ignite
> > > > OSGi-enabled
> > > > > > is
> > > > > > > the marshalling. More specifically the issue is with
> > deserialization
> > > > of
> > > > > > the
> > > > > > > classes that are provided by the bundles other than the Ignite
> > bundle
> > > > > > > itself.
> > > > > > >
> > > > > > > When the Ignite transport layer receives a message it needs to
> > figure
> > > > > out
> > > > > > > how to deserialize the bytes and for that it needs to know the
> > bundle
> > > > > > that
> > > > > > > provides the class to be deserailized. At this point TCCL is of
> > no
> > > > use.
> > > > > > To
> > > > > > > make things more complex, the class may contain other classes
> > that
> > > > come
> > > > > > > from other bundles, and so on recursively. This means that each
> > > > object
> > > > > in
> > > > > > > the hierarchy must be serialized with its bundle name (or
> bundle
> > id),
> > > > > so
> > > > > > > that the deserializer will then be able to correctly resolve
> the
> > > > class
> > > > > > > while traversing the object hierarchy during deserialization.
> > > > > > >
> > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the
> > ability to
> > > > > > plug
> > > > > > > in a custom class resolver. Romain's solution was to use Kryo
> > that
> > > > does
> > > > > > > provide a way to customize class resolution. It has solved the
> > > > problem
> > > > > of
> > > > > > > capturing the bundle info and he was able to successfully run
> > Ignite
> > > > > as a
> > > > > > > bundle in an OSGi container (after some repackaging and
> > inclusion of
> > > > > the
> > > > > > > manifest). But Kryo-based marshalling introduced a lot of
> > complexity
> > > > to
> > > > > > the
> > > > > > > code and incorrect use of Kryo's numerous serializers caused
> some
> > > > weird
> > > > > > > hard-to-debug issues in the Ignite core (like duplicate cache
> > entries
> > > > > due
> > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap class --
> > go
> > > > > > > figure!). Overall the Kryo-based solution is brittle and hard
> to
> > > > > > maintain.
> > > > > > >
> > > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > > 1) enhance the OptimizedMarshaller to allow custom class
> > resolution.
> > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to
> > the
> > > > > > > original one) to be used in OSGi environment.
> > > > > > >
> > > > > > > Regards
> > > > > > > Andrey
> > > > > > >
> > > > > > > > From: raulk@apache.org
> > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > > > To: dev@ignite.apache.org
> > > > > > > >
> > > > > > > > Hi Romain,
> > > > > > > >
> > > > > > > > I'm working on the OSGi compatibility of Ignite. I appreciate
> > your
> > > > > > input.
> > > > > > > >
> > > > > > > > I'm thinking about the situation you describe and I wonder if
> > > > you're
> > > > > > > > exporting Ignite as an OSGi service which is then consumed
> from
> > > > other
> > > > > > > > bundles. Under this situation, it would be quite easy to
> > reproduce
> > > > > the
> > > > > > > > behaviour you describe if Ignite is not resolving classes via
> > the
> > > > > TCCL.
> > > > > > > > Need to dig deeper into that.
> > > > > > > >
> > > > > > > > Off the top of my head, there are two alternatives to solve
> it:
> > > > > > > >
> > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already
> > > > used) –
> > > > > > we
> > > > > > > > gotta be wary of possible regressions.
> > > > > > > > 2. Create a special OSGi header Ignite-Export-Package so that
> > > > bundles
> > > > > > > > containing DTOs can expose packages to Ignite's marshallers.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > *Raúl Kripalani*
> > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration,
> > Big
> > > > Data
> > > > > > and
> > > > > > > > Messaging Engineer
> > > > > > > > http://about.me/raulkripalani |
> > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > > >
> > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > > romain.gilles@misys.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I'm really interested in this issue:
> > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We
> some
> > > > stuff
> > > > > to
> > > > > > > make
> > > > > > > > > it work in our osgi environment. The main issue for us now
> > is the
> > > > > > > > > serialization. I think it you will have to rework the
> > > > > > > OptimizedMarshaller
> > > > > > > > > or any other marshaller that works with object that come
> from
> > > > > outside
> > > > > > > your
> > > > > > > > > class space.
> > > > > > > > >
> > > > > > > > > We have try kryo that works. Kryo provide an extension
> point
> > in
> > > > > order
> > > > > > > to
> > > > > > > > > resolve the classes:
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > > . With this extension we are able to solve the problem of
> > > > external
> > > > > > > classes.
> > > > > > > > > The only issue with kryo is that some classes need a
> certain
> > care
> > > > > in
> > > > > > > the
> > > > > > > > > serialization process and therefore a specialized
> serializer.
> > > > > > > > >
> > > > > > > > > So I would like to know from the community what do think of
> > > > > changing
> > > > > > > the
> > > > > > > > > way the optimized marshaller works or introducing the
> > support of
> > > > > yet
> > > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks in advance,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Romain.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > > "Misys" is the trade name of the Misys group of companies.
> > This
> > > > > email
> > > > > > > and
> > > > > > > > > any attachments have been scanned for known viruses using
> > > > multiple
> > > > > > > > > scanners. This email message is intended for the named
> > recipient
> > > > > > only.
> > > > > > > It
> > > > > > > > > may be privileged and/or confidential. If you are not the
> > named
> > > > > > > recipient
> > > > > > > > > of this email please notify us immediately and do not copy
> > it or
> > > > > use
> > > > > > > it for
> > > > > > > > > any purpose, nor disclose its contents to any other person.
> > This
> > > > > > email
> > > > > > > does
> > > > > > > > > not constitute the commencement of legal relations between
> > you
> > > > and
> > > > > > > Misys.
> > > > > > > > > Please refer to the executed contract between you and the
> > > > relevant
> > > > > > > member
> > > > > > > > > of the Misys group for the identity of the contracting
> party
> > with
> > > > > > > which you
> > > > > > > > > are dealing.
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Romain Gilles <ro...@gmail.com>.
Hi,
Maybe a missing point. I think but I'm not sure that in the
OptimizedMarshaller there is a caching of already serialized classes. Due
to the dynamic nature of OSGi this may lead to memory leak. In fact if a
bundle is refreshed, it will produce a new BundleRevision and therefore a
new classloader. And if you don't release the class from the previous
BundleRevision then you endup with memory leak. So maybe the Marshaller
interface or somewhere should provide a way to free those classes /
classloaders.
Regards,
Romain
Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <an...@hotmail.com> a
écrit :
> Romain/Dmitriy,
>
> I prefer Romain's approach, but just curious, in the API you guys are
> proposing why use a byte[] rather than OutputStream/InputStream? With a
> byte[], one would inevitably end up wrapping it into a byte stream class.
> Also, the stream-based interface would be more in line with the Marshaller
> API.
>
> Also for symmetry with the readClass() method, I suggest the writeClass()
> take a Class<?> rather than an object.
>
> Regards
> Andrey
>
> > From: romain.gilles@gmail.com
> > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > Subject: Re: OSGi migration may require a special marshaller
> > To: dev@ignite.apache.org
> >
> > Hi Dmitriy,
> > I think your solution is good. Maybe I will change it a little bit... :P
> > I think you should delegate the Class resolution to the resolver. Because
> > for example with your current solution the marshaller may before (or
> after)
> > store the fqn of the class (maybe only the first time it encounter it)
> but
> > in order to save the classloader context resolution the implementation of
> > the resolver may have to save the package name of the given object and
> some
> > extra info therefore the content package name will be serialized two
> times.
> > Ok, it's not a big deal.
> > But now if the resolver identify that it can reuse the same class loader
> > for a couple of classes. It will be interesting for it to have a
> > serialization context in order to save, let say, classloader/id mapping
> in
> > order to save the id instead of a more longer representation.
> > So I propose something like that:
> > *public interface ClassResolver {*
> > * // This method is invoked on the sender side to *
> > * // marshal the information about the class.*
> > * // where the context is a map style object that is reset/new for
> each
> > object graph serialization.*
> > * public byte[] writeClass(Object o, Context context) throws
> > IgniteException;*
> >
> > * // This method is invoked on the receiving side to*
> > * // retrieve the class based on provided information.*
> > * // where the context is a map style object that is reset/new for
> each
> > object graph serialization.*
> > * public Class<?> readClass(byte[], Context context) throws
> > IgniteException;*
> > *}*
> >
> > I think your proposal will solve our issue and maybe also open a door for
> > the osgi development.
> > Let me know what do you think about me proposal? :)
> >
> > Thanks,
> >
> > Romain
> >
> > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <ds...@apache.org> a
> > écrit :
> >
> > > Romain,
> > >
> > > Can you comment on the ClassLoaderResolver suggestion that I proposed
> > > earlier? Will it solve your problem?
> > >
> > > D.
> > >
> > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <romain.gilles@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Raul,
> > > > - Do you plan to use the TCCL when marshalling in OSGi env? If yes
> how?
> > > > - I like the point 2. Maybe for a graph of object that come from
> > > different
> > > > packages / bundles you may have to recursively capture the package
> > > version
> > > > of the individual element of your graph.
> > > > - For the point 3 I wonder if it will not over-complexify the
> solution.
> > > As
> > > > a developer you will have to think about it. And it is not obvious
> in the
> > > > code where things are serialized or not. You may use lambda in your
> > > > application code therefore the current package become what you call
> a DTO
> > > > package. The current state of ignite does not enforce you to specify
> it
> > > for
> > > > "classical" classloading application. If you introduce this extra
> step
> > > for
> > > > OSGi ready application you will maybe discourage people to use OSGi.
> > > >
> > > > To comeback to our solution. We start we a strong assumption: our
> cluster
> > > > is homogeneous in term of code so, of course, it simplify our life
> :).
> > > >
> > > > BTW if you can introduce an extension point in the class resolution
> > > > mechanism it can be interesting for end user in order to allow them
> to
> > > > optimize it based on there specific use cases.
> > > >
> > > > Romain.
> > > >
> > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <ra...@evosent.com> a
> écrit :
> > > >
> > > > > Hi Andrey,
> > > > >
> > > > > Thanks for the participation in this topic.
> > > > >
> > > > > I don't like the solution to incorporate the bundle symbolic name
> in
> > > the
> > > > > serialised form. Nothing guarantees that the classdef will be
> located
> > > > under
> > > > > the same bundle in both source and target machines. We also have to
> > > take
> > > > > into account that OSGi allows for multiple versions of the same
> > > > > bundle/packages to co-exist in the same container. So it becomes
> more
> > > > > complex.
> > > > >
> > > > > Using the TCCL should work when serialising, but it'll probably be
> of
> > > no
> > > > > use when deserialising on the other end.
> > > > >
> > > > > I need to enhance Ignite to:
> > > > >
> > > > > 1. Use the TCCL when marshalling on one end.
> > > > > 2. Incorporate the package version of the class in the serialised
> form
> > > if
> > > > > Ignite is running in an OSGi environment.
> > > > > 3. On the receiver end, discover cache entities / DTOs in all
> bundles
> > > > > through a custom OSGi manifest header or the like, as I explained
> > > before.
> > > > > Package version must be taken into account.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Raúl.
> > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <an...@hotmail.com>
> wrote:
> > > > >
> > > > > > Raul,
> > > > > >
> > > > > > The fundamental hurdle we need to jump over to make Ignite
> > > OSGi-enabled
> > > > > is
> > > > > > the marshalling. More specifically the issue is with
> deserialization
> > > of
> > > > > the
> > > > > > classes that are provided by the bundles other than the Ignite
> bundle
> > > > > > itself.
> > > > > >
> > > > > > When the Ignite transport layer receives a message it needs to
> figure
> > > > out
> > > > > > how to deserialize the bytes and for that it needs to know the
> bundle
> > > > > that
> > > > > > provides the class to be deserailized. At this point TCCL is of
> no
> > > use.
> > > > > To
> > > > > > make things more complex, the class may contain other classes
> that
> > > come
> > > > > > from other bundles, and so on recursively. This means that each
> > > object
> > > > in
> > > > > > the hierarchy must be serialized with its bundle name (or bundle
> id),
> > > > so
> > > > > > that the deserializer will then be able to correctly resolve the
> > > class
> > > > > > while traversing the object hierarchy during deserialization.
> > > > > >
> > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the
> ability to
> > > > > plug
> > > > > > in a custom class resolver. Romain's solution was to use Kryo
> that
> > > does
> > > > > > provide a way to customize class resolution. It has solved the
> > > problem
> > > > of
> > > > > > capturing the bundle info and he was able to successfully run
> Ignite
> > > > as a
> > > > > > bundle in an OSGi container (after some repackaging and
> inclusion of
> > > > the
> > > > > > manifest). But Kryo-based marshalling introduced a lot of
> complexity
> > > to
> > > > > the
> > > > > > code and incorrect use of Kryo's numerous serializers caused some
> > > weird
> > > > > > hard-to-debug issues in the Ignite core (like duplicate cache
> entries
> > > > due
> > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap class --
> go
> > > > > > figure!). Overall the Kryo-based solution is brittle and hard to
> > > > > maintain.
> > > > > >
> > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > 1) enhance the OptimizedMarshaller to allow custom class
> resolution.
> > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to
> the
> > > > > > original one) to be used in OSGi environment.
> > > > > >
> > > > > > Regards
> > > > > > Andrey
> > > > > >
> > > > > > > From: raulk@apache.org
> > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > > To: dev@ignite.apache.org
> > > > > > >
> > > > > > > Hi Romain,
> > > > > > >
> > > > > > > I'm working on the OSGi compatibility of Ignite. I appreciate
> your
> > > > > input.
> > > > > > >
> > > > > > > I'm thinking about the situation you describe and I wonder if
> > > you're
> > > > > > > exporting Ignite as an OSGi service which is then consumed from
> > > other
> > > > > > > bundles. Under this situation, it would be quite easy to
> reproduce
> > > > the
> > > > > > > behaviour you describe if Ignite is not resolving classes via
> the
> > > > TCCL.
> > > > > > > Need to dig deeper into that.
> > > > > > >
> > > > > > > Off the top of my head, there are two alternatives to solve it:
> > > > > > >
> > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already
> > > used) –
> > > > > we
> > > > > > > gotta be wary of possible regressions.
> > > > > > > 2. Create a special OSGi header Ignite-Export-Package so that
> > > bundles
> > > > > > > containing DTOs can expose packages to Ignite's marshallers.
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > *Raúl Kripalani*
> > > > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration,
> Big
> > > Data
> > > > > and
> > > > > > > Messaging Engineer
> > > > > > > http://about.me/raulkripalani |
> > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > >
> > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > romain.gilles@misys.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I'm really interested in this issue:
> > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some
> > > stuff
> > > > to
> > > > > > make
> > > > > > > > it work in our osgi environment. The main issue for us now
> is the
> > > > > > > > serialization. I think it you will have to rework the
> > > > > > OptimizedMarshaller
> > > > > > > > or any other marshaller that works with object that come from
> > > > outside
> > > > > > your
> > > > > > > > class space.
> > > > > > > >
> > > > > > > > We have try kryo that works. Kryo provide an extension point
> in
> > > > order
> > > > > > to
> > > > > > > > resolve the classes:
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > . With this extension we are able to solve the problem of
> > > external
> > > > > > classes.
> > > > > > > > The only issue with kryo is that some classes need a certain
> care
> > > > in
> > > > > > the
> > > > > > > > serialization process and therefore a specialized serializer.
> > > > > > > >
> > > > > > > > So I would like to know from the community what do think of
> > > > changing
> > > > > > the
> > > > > > > > way the optimized marshaller works or introducing the
> support of
> > > > yet
> > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks in advance,
> > > > > > > >
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > >
> > > > > > > > Romain.
> > > > > > > >
> > > > > > > >
> > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > "Misys" is the trade name of the Misys group of companies.
> This
> > > > email
> > > > > > and
> > > > > > > > any attachments have been scanned for known viruses using
> > > multiple
> > > > > > > > scanners. This email message is intended for the named
> recipient
> > > > > only.
> > > > > > It
> > > > > > > > may be privileged and/or confidential. If you are not the
> named
> > > > > > recipient
> > > > > > > > of this email please notify us immediately and do not copy
> it or
> > > > use
> > > > > > it for
> > > > > > > > any purpose, nor disclose its contents to any other person.
> This
> > > > > email
> > > > > > does
> > > > > > > > not constitute the commencement of legal relations between
> you
> > > and
> > > > > > Misys.
> > > > > > > > Please refer to the executed contract between you and the
> > > relevant
> > > > > > member
> > > > > > > > of the Misys group for the identity of the contracting party
> with
> > > > > > which you
> > > > > > > > are dealing.
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
RE: OSGi migration may require a special marshaller
Posted by Andrey Kornev <an...@hotmail.com>.
Romain/Dmitriy,
I prefer Romain's approach, but just curious, in the API you guys are proposing why use a byte[] rather than OutputStream/InputStream? With a byte[], one would inevitably end up wrapping it into a byte stream class. Also, the stream-based interface would be more in line with the Marshaller API.
Also for symmetry with the readClass() method, I suggest the writeClass() take a Class<?> rather than an object.
Regards
Andrey
> From: romain.gilles@gmail.com
> Date: Tue, 3 Nov 2015 21:24:01 +0000
> Subject: Re: OSGi migration may require a special marshaller
> To: dev@ignite.apache.org
>
> Hi Dmitriy,
> I think your solution is good. Maybe I will change it a little bit... :P
> I think you should delegate the Class resolution to the resolver. Because
> for example with your current solution the marshaller may before (or after)
> store the fqn of the class (maybe only the first time it encounter it) but
> in order to save the classloader context resolution the implementation of
> the resolver may have to save the package name of the given object and some
> extra info therefore the content package name will be serialized two times.
> Ok, it's not a big deal.
> But now if the resolver identify that it can reuse the same class loader
> for a couple of classes. It will be interesting for it to have a
> serialization context in order to save, let say, classloader/id mapping in
> order to save the id instead of a more longer representation.
> So I propose something like that:
> *public interface ClassResolver {*
> * // This method is invoked on the sender side to *
> * // marshal the information about the class.*
> * // where the context is a map style object that is reset/new for each
> object graph serialization.*
> * public byte[] writeClass(Object o, Context context) throws
> IgniteException;*
>
> * // This method is invoked on the receiving side to*
> * // retrieve the class based on provided information.*
> * // where the context is a map style object that is reset/new for each
> object graph serialization.*
> * public Class<?> readClass(byte[], Context context) throws
> IgniteException;*
> *}*
>
> I think your proposal will solve our issue and maybe also open a door for
> the osgi development.
> Let me know what do you think about me proposal? :)
>
> Thanks,
>
> Romain
>
> Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <ds...@apache.org> a
> écrit :
>
> > Romain,
> >
> > Can you comment on the ClassLoaderResolver suggestion that I proposed
> > earlier? Will it solve your problem?
> >
> > D.
> >
> > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <ro...@gmail.com>
> > wrote:
> >
> > > Hi Raul,
> > > - Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
> > > - I like the point 2. Maybe for a graph of object that come from
> > different
> > > packages / bundles you may have to recursively capture the package
> > version
> > > of the individual element of your graph.
> > > - For the point 3 I wonder if it will not over-complexify the solution.
> > As
> > > a developer you will have to think about it. And it is not obvious in the
> > > code where things are serialized or not. You may use lambda in your
> > > application code therefore the current package become what you call a DTO
> > > package. The current state of ignite does not enforce you to specify it
> > for
> > > "classical" classloading application. If you introduce this extra step
> > for
> > > OSGi ready application you will maybe discourage people to use OSGi.
> > >
> > > To comeback to our solution. We start we a strong assumption: our cluster
> > > is homogeneous in term of code so, of course, it simplify our life :).
> > >
> > > BTW if you can introduce an extension point in the class resolution
> > > mechanism it can be interesting for end user in order to allow them to
> > > optimize it based on there specific use cases.
> > >
> > > Romain.
> > >
> > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <ra...@evosent.com> a écrit :
> > >
> > > > Hi Andrey,
> > > >
> > > > Thanks for the participation in this topic.
> > > >
> > > > I don't like the solution to incorporate the bundle symbolic name in
> > the
> > > > serialised form. Nothing guarantees that the classdef will be located
> > > under
> > > > the same bundle in both source and target machines. We also have to
> > take
> > > > into account that OSGi allows for multiple versions of the same
> > > > bundle/packages to co-exist in the same container. So it becomes more
> > > > complex.
> > > >
> > > > Using the TCCL should work when serialising, but it'll probably be of
> > no
> > > > use when deserialising on the other end.
> > > >
> > > > I need to enhance Ignite to:
> > > >
> > > > 1. Use the TCCL when marshalling on one end.
> > > > 2. Incorporate the package version of the class in the serialised form
> > if
> > > > Ignite is running in an OSGi environment.
> > > > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > > > through a custom OSGi manifest header or the like, as I explained
> > before.
> > > > Package version must be taken into account.
> > > >
> > > > What do you think?
> > > >
> > > > Raúl.
> > > > On 2 Nov 2015 17:25, "Andrey Kornev" <an...@hotmail.com> wrote:
> > > >
> > > > > Raul,
> > > > >
> > > > > The fundamental hurdle we need to jump over to make Ignite
> > OSGi-enabled
> > > > is
> > > > > the marshalling. More specifically the issue is with deserialization
> > of
> > > > the
> > > > > classes that are provided by the bundles other than the Ignite bundle
> > > > > itself.
> > > > >
> > > > > When the Ignite transport layer receives a message it needs to figure
> > > out
> > > > > how to deserialize the bytes and for that it needs to know the bundle
> > > > that
> > > > > provides the class to be deserailized. At this point TCCL is of no
> > use.
> > > > To
> > > > > make things more complex, the class may contain other classes that
> > come
> > > > > from other bundles, and so on recursively. This means that each
> > object
> > > in
> > > > > the hierarchy must be serialized with its bundle name (or bundle id),
> > > so
> > > > > that the deserializer will then be able to correctly resolve the
> > class
> > > > > while traversing the object hierarchy during deserialization.
> > > > >
> > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> > > > plug
> > > > > in a custom class resolver. Romain's solution was to use Kryo that
> > does
> > > > > provide a way to customize class resolution. It has solved the
> > problem
> > > of
> > > > > capturing the bundle info and he was able to successfully run Ignite
> > > as a
> > > > > bundle in an OSGi container (after some repackaging and inclusion of
> > > the
> > > > > manifest). But Kryo-based marshalling introduced a lot of complexity
> > to
> > > > the
> > > > > code and incorrect use of Kryo's numerous serializers caused some
> > weird
> > > > > hard-to-debug issues in the Ignite core (like duplicate cache entries
> > > due
> > > > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > > > > figure!). Overall the Kryo-based solution is brittle and hard to
> > > > maintain.
> > > > >
> > > > > I feel the correct solution to OSGi problem would be to
> > > > > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > > > > original one) to be used in OSGi environment.
> > > > >
> > > > > Regards
> > > > > Andrey
> > > > >
> > > > > > From: raulk@apache.org
> > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > To: dev@ignite.apache.org
> > > > > >
> > > > > > Hi Romain,
> > > > > >
> > > > > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> > > > input.
> > > > > >
> > > > > > I'm thinking about the situation you describe and I wonder if
> > you're
> > > > > > exporting Ignite as an OSGi service which is then consumed from
> > other
> > > > > > bundles. Under this situation, it would be quite easy to reproduce
> > > the
> > > > > > behaviour you describe if Ignite is not resolving classes via the
> > > TCCL.
> > > > > > Need to dig deeper into that.
> > > > > >
> > > > > > Off the top of my head, there are two alternatives to solve it:
> > > > > >
> > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already
> > used) –
> > > > we
> > > > > > gotta be wary of possible regressions.
> > > > > > 2. Create a special OSGi header Ignite-Export-Package so that
> > bundles
> > > > > > containing DTOs can expose packages to Ignite's marshallers.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > *Raúl Kripalani*
> > > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> > Data
> > > > and
> > > > > > Messaging Engineer
> > > > > > http://about.me/raulkripalani |
> > > > http://www.linkedin.com/in/raulkripalani
> > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > >
> > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > romain.gilles@misys.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'm really interested in this issue:
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some
> > stuff
> > > to
> > > > > make
> > > > > > > it work in our osgi environment. The main issue for us now is the
> > > > > > > serialization. I think it you will have to rework the
> > > > > OptimizedMarshaller
> > > > > > > or any other marshaller that works with object that come from
> > > outside
> > > > > your
> > > > > > > class space.
> > > > > > >
> > > > > > > We have try kryo that works. Kryo provide an extension point in
> > > order
> > > > > to
> > > > > > > resolve the classes:
> > > > > > >
> > > > >
> > > >
> > >
> > https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > . With this extension we are able to solve the problem of
> > external
> > > > > classes.
> > > > > > > The only issue with kryo is that some classes need a certain care
> > > in
> > > > > the
> > > > > > > serialization process and therefore a specialized serializer.
> > > > > > >
> > > > > > > So I would like to know from the community what do think of
> > > changing
> > > > > the
> > > > > > > way the optimized marshaller works or introducing the support of
> > > yet
> > > > > > > another marshaller based on a kryo like technology?
> > > > > > >
> > > > > > >
> > > > > > > Thanks in advance,
> > > > > > >
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > >
> > > > > > > Romain.
> > > > > > >
> > > > > > >
> > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > "Misys" is the trade name of the Misys group of companies. This
> > > email
> > > > > and
> > > > > > > any attachments have been scanned for known viruses using
> > multiple
> > > > > > > scanners. This email message is intended for the named recipient
> > > > only.
> > > > > It
> > > > > > > may be privileged and/or confidential. If you are not the named
> > > > > recipient
> > > > > > > of this email please notify us immediately and do not copy it or
> > > use
> > > > > it for
> > > > > > > any purpose, nor disclose its contents to any other person. This
> > > > email
> > > > > does
> > > > > > > not constitute the commencement of legal relations between you
> > and
> > > > > Misys.
> > > > > > > Please refer to the executed contract between you and the
> > relevant
> > > > > member
> > > > > > > of the Misys group for the identity of the contracting party with
> > > > > which you
> > > > > > > are dealing.
> > > > > > >
> > > > >
> > > >
> > >
> >
Re: OSGi migration may require a special marshaller
Posted by Romain Gilles <ro...@gmail.com>.
Hi Dmitriy,
I think your solution is good. Maybe I will change it a little bit... :P
I think you should delegate the Class resolution to the resolver. Because
for example with your current solution the marshaller may before (or after)
store the fqn of the class (maybe only the first time it encounter it) but
in order to save the classloader context resolution the implementation of
the resolver may have to save the package name of the given object and some
extra info therefore the content package name will be serialized two times.
Ok, it's not a big deal.
But now if the resolver identify that it can reuse the same class loader
for a couple of classes. It will be interesting for it to have a
serialization context in order to save, let say, classloader/id mapping in
order to save the id instead of a more longer representation.
So I propose something like that:
*public interface ClassResolver {*
* // This method is invoked on the sender side to *
* // marshal the information about the class.*
* // where the context is a map style object that is reset/new for each
object graph serialization.*
* public byte[] writeClass(Object o, Context context) throws
IgniteException;*
* // This method is invoked on the receiving side to*
* // retrieve the class based on provided information.*
* // where the context is a map style object that is reset/new for each
object graph serialization.*
* public Class<?> readClass(byte[], Context context) throws
IgniteException;*
*}*
I think your proposal will solve our issue and maybe also open a door for
the osgi development.
Let me know what do you think about me proposal? :)
Thanks,
Romain
Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <ds...@apache.org> a
écrit :
> Romain,
>
> Can you comment on the ClassLoaderResolver suggestion that I proposed
> earlier? Will it solve your problem?
>
> D.
>
> On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <ro...@gmail.com>
> wrote:
>
> > Hi Raul,
> > - Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
> > - I like the point 2. Maybe for a graph of object that come from
> different
> > packages / bundles you may have to recursively capture the package
> version
> > of the individual element of your graph.
> > - For the point 3 I wonder if it will not over-complexify the solution.
> As
> > a developer you will have to think about it. And it is not obvious in the
> > code where things are serialized or not. You may use lambda in your
> > application code therefore the current package become what you call a DTO
> > package. The current state of ignite does not enforce you to specify it
> for
> > "classical" classloading application. If you introduce this extra step
> for
> > OSGi ready application you will maybe discourage people to use OSGi.
> >
> > To comeback to our solution. We start we a strong assumption: our cluster
> > is homogeneous in term of code so, of course, it simplify our life :).
> >
> > BTW if you can introduce an extension point in the class resolution
> > mechanism it can be interesting for end user in order to allow them to
> > optimize it based on there specific use cases.
> >
> > Romain.
> >
> > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <ra...@evosent.com> a écrit :
> >
> > > Hi Andrey,
> > >
> > > Thanks for the participation in this topic.
> > >
> > > I don't like the solution to incorporate the bundle symbolic name in
> the
> > > serialised form. Nothing guarantees that the classdef will be located
> > under
> > > the same bundle in both source and target machines. We also have to
> take
> > > into account that OSGi allows for multiple versions of the same
> > > bundle/packages to co-exist in the same container. So it becomes more
> > > complex.
> > >
> > > Using the TCCL should work when serialising, but it'll probably be of
> no
> > > use when deserialising on the other end.
> > >
> > > I need to enhance Ignite to:
> > >
> > > 1. Use the TCCL when marshalling on one end.
> > > 2. Incorporate the package version of the class in the serialised form
> if
> > > Ignite is running in an OSGi environment.
> > > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > > through a custom OSGi manifest header or the like, as I explained
> before.
> > > Package version must be taken into account.
> > >
> > > What do you think?
> > >
> > > Raúl.
> > > On 2 Nov 2015 17:25, "Andrey Kornev" <an...@hotmail.com> wrote:
> > >
> > > > Raul,
> > > >
> > > > The fundamental hurdle we need to jump over to make Ignite
> OSGi-enabled
> > > is
> > > > the marshalling. More specifically the issue is with deserialization
> of
> > > the
> > > > classes that are provided by the bundles other than the Ignite bundle
> > > > itself.
> > > >
> > > > When the Ignite transport layer receives a message it needs to figure
> > out
> > > > how to deserialize the bytes and for that it needs to know the bundle
> > > that
> > > > provides the class to be deserailized. At this point TCCL is of no
> use.
> > > To
> > > > make things more complex, the class may contain other classes that
> come
> > > > from other bundles, and so on recursively. This means that each
> object
> > in
> > > > the hierarchy must be serialized with its bundle name (or bundle id),
> > so
> > > > that the deserializer will then be able to correctly resolve the
> class
> > > > while traversing the object hierarchy during deserialization.
> > > >
> > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> > > plug
> > > > in a custom class resolver. Romain's solution was to use Kryo that
> does
> > > > provide a way to customize class resolution. It has solved the
> problem
> > of
> > > > capturing the bundle info and he was able to successfully run Ignite
> > as a
> > > > bundle in an OSGi container (after some repackaging and inclusion of
> > the
> > > > manifest). But Kryo-based marshalling introduced a lot of complexity
> to
> > > the
> > > > code and incorrect use of Kryo's numerous serializers caused some
> weird
> > > > hard-to-debug issues in the Ignite core (like duplicate cache entries
> > due
> > > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > > > figure!). Overall the Kryo-based solution is brittle and hard to
> > > maintain.
> > > >
> > > > I feel the correct solution to OSGi problem would be to
> > > > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > > > original one) to be used in OSGi environment.
> > > >
> > > > Regards
> > > > Andrey
> > > >
> > > > > From: raulk@apache.org
> > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > To: dev@ignite.apache.org
> > > > >
> > > > > Hi Romain,
> > > > >
> > > > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> > > input.
> > > > >
> > > > > I'm thinking about the situation you describe and I wonder if
> you're
> > > > > exporting Ignite as an OSGi service which is then consumed from
> other
> > > > > bundles. Under this situation, it would be quite easy to reproduce
> > the
> > > > > behaviour you describe if Ignite is not resolving classes via the
> > TCCL.
> > > > > Need to dig deeper into that.
> > > > >
> > > > > Off the top of my head, there are two alternatives to solve it:
> > > > >
> > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already
> used) –
> > > we
> > > > > gotta be wary of possible regressions.
> > > > > 2. Create a special OSGi header Ignite-Export-Package so that
> bundles
> > > > > containing DTOs can expose packages to Ignite's marshallers.
> > > > >
> > > > > Regards,
> > > > >
> > > > > *Raúl Kripalani*
> > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> Data
> > > and
> > > > > Messaging Engineer
> > > > > http://about.me/raulkripalani |
> > > http://www.linkedin.com/in/raulkripalani
> > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > >
> > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > romain.gilles@misys.com>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'm really interested in this issue:
> > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some
> stuff
> > to
> > > > make
> > > > > > it work in our osgi environment. The main issue for us now is the
> > > > > > serialization. I think it you will have to rework the
> > > > OptimizedMarshaller
> > > > > > or any other marshaller that works with object that come from
> > outside
> > > > your
> > > > > > class space.
> > > > > >
> > > > > > We have try kryo that works. Kryo provide an extension point in
> > order
> > > > to
> > > > > > resolve the classes:
> > > > > >
> > > >
> > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > . With this extension we are able to solve the problem of
> external
> > > > classes.
> > > > > > The only issue with kryo is that some classes need a certain care
> > in
> > > > the
> > > > > > serialization process and therefore a specialized serializer.
> > > > > >
> > > > > > So I would like to know from the community what do think of
> > changing
> > > > the
> > > > > > way the optimized marshaller works or introducing the support of
> > yet
> > > > > > another marshaller based on a kryo like technology?
> > > > > >
> > > > > >
> > > > > > Thanks in advance,
> > > > > >
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > >
> > > > > > Romain.
> > > > > >
> > > > > >
> > > > > > PS: I'm ready to help in the both cases.
> > > > > > "Misys" is the trade name of the Misys group of companies. This
> > email
> > > > and
> > > > > > any attachments have been scanned for known viruses using
> multiple
> > > > > > scanners. This email message is intended for the named recipient
> > > only.
> > > > It
> > > > > > may be privileged and/or confidential. If you are not the named
> > > > recipient
> > > > > > of this email please notify us immediately and do not copy it or
> > use
> > > > it for
> > > > > > any purpose, nor disclose its contents to any other person. This
> > > email
> > > > does
> > > > > > not constitute the commencement of legal relations between you
> and
> > > > Misys.
> > > > > > Please refer to the executed contract between you and the
> relevant
> > > > member
> > > > > > of the Misys group for the identity of the contracting party with
> > > > which you
> > > > > > are dealing.
> > > > > >
> > > >
> > >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Romain,
Can you comment on the ClassLoaderResolver suggestion that I proposed
earlier? Will it solve your problem?
D.
On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <ro...@gmail.com>
wrote:
> Hi Raul,
> - Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
> - I like the point 2. Maybe for a graph of object that come from different
> packages / bundles you may have to recursively capture the package version
> of the individual element of your graph.
> - For the point 3 I wonder if it will not over-complexify the solution. As
> a developer you will have to think about it. And it is not obvious in the
> code where things are serialized or not. You may use lambda in your
> application code therefore the current package become what you call a DTO
> package. The current state of ignite does not enforce you to specify it for
> "classical" classloading application. If you introduce this extra step for
> OSGi ready application you will maybe discourage people to use OSGi.
>
> To comeback to our solution. We start we a strong assumption: our cluster
> is homogeneous in term of code so, of course, it simplify our life :).
>
> BTW if you can introduce an extension point in the class resolution
> mechanism it can be interesting for end user in order to allow them to
> optimize it based on there specific use cases.
>
> Romain.
>
> Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <ra...@evosent.com> a écrit :
>
> > Hi Andrey,
> >
> > Thanks for the participation in this topic.
> >
> > I don't like the solution to incorporate the bundle symbolic name in the
> > serialised form. Nothing guarantees that the classdef will be located
> under
> > the same bundle in both source and target machines. We also have to take
> > into account that OSGi allows for multiple versions of the same
> > bundle/packages to co-exist in the same container. So it becomes more
> > complex.
> >
> > Using the TCCL should work when serialising, but it'll probably be of no
> > use when deserialising on the other end.
> >
> > I need to enhance Ignite to:
> >
> > 1. Use the TCCL when marshalling on one end.
> > 2. Incorporate the package version of the class in the serialised form if
> > Ignite is running in an OSGi environment.
> > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > through a custom OSGi manifest header or the like, as I explained before.
> > Package version must be taken into account.
> >
> > What do you think?
> >
> > Raúl.
> > On 2 Nov 2015 17:25, "Andrey Kornev" <an...@hotmail.com> wrote:
> >
> > > Raul,
> > >
> > > The fundamental hurdle we need to jump over to make Ignite OSGi-enabled
> > is
> > > the marshalling. More specifically the issue is with deserialization of
> > the
> > > classes that are provided by the bundles other than the Ignite bundle
> > > itself.
> > >
> > > When the Ignite transport layer receives a message it needs to figure
> out
> > > how to deserialize the bytes and for that it needs to know the bundle
> > that
> > > provides the class to be deserailized. At this point TCCL is of no use.
> > To
> > > make things more complex, the class may contain other classes that come
> > > from other bundles, and so on recursively. This means that each object
> in
> > > the hierarchy must be serialized with its bundle name (or bundle id),
> so
> > > that the deserializer will then be able to correctly resolve the class
> > > while traversing the object hierarchy during deserialization.
> > >
> > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> > plug
> > > in a custom class resolver. Romain's solution was to use Kryo that does
> > > provide a way to customize class resolution. It has solved the problem
> of
> > > capturing the bundle info and he was able to successfully run Ignite
> as a
> > > bundle in an OSGi container (after some repackaging and inclusion of
> the
> > > manifest). But Kryo-based marshalling introduced a lot of complexity to
> > the
> > > code and incorrect use of Kryo's numerous serializers caused some weird
> > > hard-to-debug issues in the Ignite core (like duplicate cache entries
> due
> > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > > figure!). Overall the Kryo-based solution is brittle and hard to
> > maintain.
> > >
> > > I feel the correct solution to OSGi problem would be to
> > > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > > original one) to be used in OSGi environment.
> > >
> > > Regards
> > > Andrey
> > >
> > > > From: raulk@apache.org
> > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > Subject: Re: OSGi migration may require a special marshaller
> > > > To: dev@ignite.apache.org
> > > >
> > > > Hi Romain,
> > > >
> > > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> > input.
> > > >
> > > > I'm thinking about the situation you describe and I wonder if you're
> > > > exporting Ignite as an OSGi service which is then consumed from other
> > > > bundles. Under this situation, it would be quite easy to reproduce
> the
> > > > behaviour you describe if Ignite is not resolving classes via the
> TCCL.
> > > > Need to dig deeper into that.
> > > >
> > > > Off the top of my head, there are two alternatives to solve it:
> > > >
> > > > 1. Use the TCCL for marshalling/unmarshalling (if not already used) –
> > we
> > > > gotta be wary of possible regressions.
> > > > 2. Create a special OSGi header Ignite-Export-Package so that bundles
> > > > containing DTOs can expose packages to Ignite's marshallers.
> > > >
> > > > Regards,
> > > >
> > > > *Raúl Kripalani*
> > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> > and
> > > > Messaging Engineer
> > > > http://about.me/raulkripalani |
> > http://www.linkedin.com/in/raulkripalani
> > > > http://blog.raulkr.net | twitter: @raulvk
> > > >
> > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > romain.gilles@misys.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'm really interested in this issue:
> > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff
> to
> > > make
> > > > > it work in our osgi environment. The main issue for us now is the
> > > > > serialization. I think it you will have to rework the
> > > OptimizedMarshaller
> > > > > or any other marshaller that works with object that come from
> outside
> > > your
> > > > > class space.
> > > > >
> > > > > We have try kryo that works. Kryo provide an extension point in
> order
> > > to
> > > > > resolve the classes:
> > > > >
> > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > . With this extension we are able to solve the problem of external
> > > classes.
> > > > > The only issue with kryo is that some classes need a certain care
> in
> > > the
> > > > > serialization process and therefore a specialized serializer.
> > > > >
> > > > > So I would like to know from the community what do think of
> changing
> > > the
> > > > > way the optimized marshaller works or introducing the support of
> yet
> > > > > another marshaller based on a kryo like technology?
> > > > >
> > > > >
> > > > > Thanks in advance,
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > >
> > > > > Romain.
> > > > >
> > > > >
> > > > > PS: I'm ready to help in the both cases.
> > > > > "Misys" is the trade name of the Misys group of companies. This
> email
> > > and
> > > > > any attachments have been scanned for known viruses using multiple
> > > > > scanners. This email message is intended for the named recipient
> > only.
> > > It
> > > > > may be privileged and/or confidential. If you are not the named
> > > recipient
> > > > > of this email please notify us immediately and do not copy it or
> use
> > > it for
> > > > > any purpose, nor disclose its contents to any other person. This
> > email
> > > does
> > > > > not constitute the commencement of legal relations between you and
> > > Misys.
> > > > > Please refer to the executed contract between you and the relevant
> > > member
> > > > > of the Misys group for the identity of the contracting party with
> > > which you
> > > > > are dealing.
> > > > >
> > >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Romain Gilles <ro...@gmail.com>.
Hi Raul,
- Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
- I like the point 2. Maybe for a graph of object that come from different
packages / bundles you may have to recursively capture the package version
of the individual element of your graph.
- For the point 3 I wonder if it will not over-complexify the solution. As
a developer you will have to think about it. And it is not obvious in the
code where things are serialized or not. You may use lambda in your
application code therefore the current package become what you call a DTO
package. The current state of ignite does not enforce you to specify it for
"classical" classloading application. If you introduce this extra step for
OSGi ready application you will maybe discourage people to use OSGi.
To comeback to our solution. We start we a strong assumption: our cluster
is homogeneous in term of code so, of course, it simplify our life :).
BTW if you can introduce an extension point in the class resolution
mechanism it can be interesting for end user in order to allow them to
optimize it based on there specific use cases.
Romain.
Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <ra...@evosent.com> a écrit :
> Hi Andrey,
>
> Thanks for the participation in this topic.
>
> I don't like the solution to incorporate the bundle symbolic name in the
> serialised form. Nothing guarantees that the classdef will be located under
> the same bundle in both source and target machines. We also have to take
> into account that OSGi allows for multiple versions of the same
> bundle/packages to co-exist in the same container. So it becomes more
> complex.
>
> Using the TCCL should work when serialising, but it'll probably be of no
> use when deserialising on the other end.
>
> I need to enhance Ignite to:
>
> 1. Use the TCCL when marshalling on one end.
> 2. Incorporate the package version of the class in the serialised form if
> Ignite is running in an OSGi environment.
> 3. On the receiver end, discover cache entities / DTOs in all bundles
> through a custom OSGi manifest header or the like, as I explained before.
> Package version must be taken into account.
>
> What do you think?
>
> Raúl.
> On 2 Nov 2015 17:25, "Andrey Kornev" <an...@hotmail.com> wrote:
>
> > Raul,
> >
> > The fundamental hurdle we need to jump over to make Ignite OSGi-enabled
> is
> > the marshalling. More specifically the issue is with deserialization of
> the
> > classes that are provided by the bundles other than the Ignite bundle
> > itself.
> >
> > When the Ignite transport layer receives a message it needs to figure out
> > how to deserialize the bytes and for that it needs to know the bundle
> that
> > provides the class to be deserailized. At this point TCCL is of no use.
> To
> > make things more complex, the class may contain other classes that come
> > from other bundles, and so on recursively. This means that each object in
> > the hierarchy must be serialized with its bundle name (or bundle id), so
> > that the deserializer will then be able to correctly resolve the class
> > while traversing the object hierarchy during deserialization.
> >
> > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> plug
> > in a custom class resolver. Romain's solution was to use Kryo that does
> > provide a way to customize class resolution. It has solved the problem of
> > capturing the bundle info and he was able to successfully run Ignite as a
> > bundle in an OSGi container (after some repackaging and inclusion of the
> > manifest). But Kryo-based marshalling introduced a lot of complexity to
> the
> > code and incorrect use of Kryo's numerous serializers caused some weird
> > hard-to-debug issues in the Ignite core (like duplicate cache entries due
> > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > figure!). Overall the Kryo-based solution is brittle and hard to
> maintain.
> >
> > I feel the correct solution to OSGi problem would be to
> > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > original one) to be used in OSGi environment.
> >
> > Regards
> > Andrey
> >
> > > From: raulk@apache.org
> > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: dev@ignite.apache.org
> > >
> > > Hi Romain,
> > >
> > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> input.
> > >
> > > I'm thinking about the situation you describe and I wonder if you're
> > > exporting Ignite as an OSGi service which is then consumed from other
> > > bundles. Under this situation, it would be quite easy to reproduce the
> > > behaviour you describe if Ignite is not resolving classes via the TCCL.
> > > Need to dig deeper into that.
> > >
> > > Off the top of my head, there are two alternatives to solve it:
> > >
> > > 1. Use the TCCL for marshalling/unmarshalling (if not already used) –
> we
> > > gotta be wary of possible regressions.
> > > 2. Create a special OSGi header Ignite-Export-Package so that bundles
> > > containing DTOs can expose packages to Ignite's marshallers.
> > >
> > > Regards,
> > >
> > > *Raúl Kripalani*
> > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> and
> > > Messaging Engineer
> > > http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> > > http://blog.raulkr.net | twitter: @raulvk
> > >
> > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> romain.gilles@misys.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm really interested in this issue:
> > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to
> > make
> > > > it work in our osgi environment. The main issue for us now is the
> > > > serialization. I think it you will have to rework the
> > OptimizedMarshaller
> > > > or any other marshaller that works with object that come from outside
> > your
> > > > class space.
> > > >
> > > > We have try kryo that works. Kryo provide an extension point in order
> > to
> > > > resolve the classes:
> > > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > . With this extension we are able to solve the problem of external
> > classes.
> > > > The only issue with kryo is that some classes need a certain care in
> > the
> > > > serialization process and therefore a specialized serializer.
> > > >
> > > > So I would like to know from the community what do think of changing
> > the
> > > > way the optimized marshaller works or introducing the support of yet
> > > > another marshaller based on a kryo like technology?
> > > >
> > > >
> > > > Thanks in advance,
> > > >
> > > >
> > > > Best regards,
> > > >
> > > >
> > > > Romain.
> > > >
> > > >
> > > > PS: I'm ready to help in the both cases.
> > > > "Misys" is the trade name of the Misys group of companies. This email
> > and
> > > > any attachments have been scanned for known viruses using multiple
> > > > scanners. This email message is intended for the named recipient
> only.
> > It
> > > > may be privileged and/or confidential. If you are not the named
> > recipient
> > > > of this email please notify us immediately and do not copy it or use
> > it for
> > > > any purpose, nor disclose its contents to any other person. This
> email
> > does
> > > > not constitute the commencement of legal relations between you and
> > Misys.
> > > > Please refer to the executed contract between you and the relevant
> > member
> > > > of the Misys group for the identity of the contracting party with
> > which you
> > > > are dealing.
> > > >
> >
>
Re: OSGi migration may require a special marshaller
Posted by Dmitriy Setrakyan <ds...@apache.org>.
I think I can see Andrey’s point about the bundles. Having same classdef in
different bundles on sender and receiver sides sounds rather like a bug,
not a feature we should support.
Having read this thread, I think we need to have ClassLoaderResolver on the
sending and receiving sides. Something like this:
*public interface ClassLoaderResolver {*
* // This method is invoked on the sender side to *
* // marshal the information about the class loader.*
* public byte[] writeClassLoader(Object o) throws IgniteException;*
* // This method is invoked on the receiving side to*
* // retrieve the class loader based on provided information.*
* public ClassLoader readClassLoader(byte[]) throws IgniteException;*
*}*
This resolver should be a pluggable configuration for marshallers or
provided in IgniteConfiguration directly. Once we have such resolver, then
we can have a special OSGI implementation for it (perhaps a different one
for different OSGI containers).
Thoughts?
D.
On Mon, Nov 2, 2015 at 4:23 PM, Andrey Kornev <an...@hotmail.com>
wrote:
> Hey Raul,
>
> What kind of use case do you have in mind when you suggest that a classdef
> may be located in different bundles on different machines? What would one
> have to do to end up in such a predicament? It feels more like a
> configuration/deployment issue rather than a desirable feature.
>
> A more realistic assumption I believe is that the nodes will have (mostly)
> the same set of bundles deployed. (We even enforce it at runtime by
> leveraging Ignite's PluginProvider.validateNewNode() method: we do not
> allow new nodes join the cluster if their OSGi environment is inconsistent.)
>
> The "bundle name" in my original email really meant "bundle symbolic name
> + bundle version". Sorry for not being clear.
>
> I'm not sure I understand why TCCL is even required for serialization?
>
> Finally, I didn't quite grok your thinking about the custom manifest
> header. Could you please elaborate? What type of information do you propose
> to store in the header? How will the serializer be able to provide that
> info, and how will the deserializer use the info to load the classes? Also,
> who is responsible for making sure that the custom header includes all
> classes to be serialized? And how/when/by whom would such a header be
> generated?
>
> Regards
> Andrey
>
> > Date: Mon, 2 Nov 2015 23:20:41 +0000
> > Subject: RE: OSGi migration may require a special marshaller
> > From: raul@evosent.com
> > To: dev@ignite.apache.org
> >
> > Hi Andrey,
> >
> > Thanks for the participation in this topic.
> >
> > I don't like the solution to incorporate the bundle symbolic name in the
> > serialised form. Nothing guarantees that the classdef will be located
> under
> > the same bundle in both source and target machines. We also have to take
> > into account that OSGi allows for multiple versions of the same
> > bundle/packages to co-exist in the same container. So it becomes more
> > complex.
> >
> > Using the TCCL should work when serialising, but it'll probably be of no
> > use when deserialising on the other end.
> >
> > I need to enhance Ignite to:
> >
> > 1. Use the TCCL when marshalling on one end.
> > 2. Incorporate the package version of the class in the serialised form if
> > Ignite is running in an OSGi environment.
> > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > through a custom OSGi manifest header or the like, as I explained before.
> > Package version must be taken into account.
> >
> > What do you think?
> >
> > Raúl.
> > On 2 Nov 2015 17:25, "Andrey Kornev" <an...@hotmail.com> wrote:
> >
> > > Raul,
> > >
> > > The fundamental hurdle we need to jump over to make Ignite
> OSGi-enabled is
> > > the marshalling. More specifically the issue is with deserialization
> of the
> > > classes that are provided by the bundles other than the Ignite bundle
> > > itself.
> > >
> > > When the Ignite transport layer receives a message it needs to figure
> out
> > > how to deserialize the bytes and for that it needs to know the bundle
> that
> > > provides the class to be deserailized. At this point TCCL is of no
> use. To
> > > make things more complex, the class may contain other classes that come
> > > from other bundles, and so on recursively. This means that each object
> in
> > > the hierarchy must be serialized with its bundle name (or bundle id),
> so
> > > that the deserializer will then be able to correctly resolve the class
> > > while traversing the object hierarchy during deserialization.
> > >
> > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> plug
> > > in a custom class resolver. Romain's solution was to use Kryo that does
> > > provide a way to customize class resolution. It has solved the problem
> of
> > > capturing the bundle info and he was able to successfully run Ignite
> as a
> > > bundle in an OSGi container (after some repackaging and inclusion of
> the
> > > manifest). But Kryo-based marshalling introduced a lot of complexity
> to the
> > > code and incorrect use of Kryo's numerous serializers caused some weird
> > > hard-to-debug issues in the Ignite core (like duplicate cache entries
> due
> > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > > figure!). Overall the Kryo-based solution is brittle and hard to
> maintain.
> > >
> > > I feel the correct solution to OSGi problem would be to
> > > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > > original one) to be used in OSGi environment.
> > >
> > > Regards
> > > Andrey
> > >
> > > > From: raulk@apache.org
> > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > Subject: Re: OSGi migration may require a special marshaller
> > > > To: dev@ignite.apache.org
> > > >
> > > > Hi Romain,
> > > >
> > > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> input.
> > > >
> > > > I'm thinking about the situation you describe and I wonder if you're
> > > > exporting Ignite as an OSGi service which is then consumed from other
> > > > bundles. Under this situation, it would be quite easy to reproduce
> the
> > > > behaviour you describe if Ignite is not resolving classes via the
> TCCL.
> > > > Need to dig deeper into that.
> > > >
> > > > Off the top of my head, there are two alternatives to solve it:
> > > >
> > > > 1. Use the TCCL for marshalling/unmarshalling (if not already used)
> – we
> > > > gotta be wary of possible regressions.
> > > > 2. Create a special OSGi header Ignite-Export-Package so that bundles
> > > > containing DTOs can expose packages to Ignite's marshallers.
> > > >
> > > > Regards,
> > > >
> > > > *Raúl Kripalani*
> > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> Data and
> > > > Messaging Engineer
> > > > http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> > > > http://blog.raulkr.net | twitter: @raulvk
> > > >
> > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> romain.gilles@misys.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'm really interested in this issue:
> > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff
> to
> > > make
> > > > > it work in our osgi environment. The main issue for us now is the
> > > > > serialization. I think it you will have to rework the
> > > OptimizedMarshaller
> > > > > or any other marshaller that works with object that come from
> outside
> > > your
> > > > > class space.
> > > > >
> > > > > We have try kryo that works. Kryo provide an extension point in
> order
> > > to
> > > > > resolve the classes:
> > > > >
> > >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > . With this extension we are able to solve the problem of external
> > > classes.
> > > > > The only issue with kryo is that some classes need a certain care
> in
> > > the
> > > > > serialization process and therefore a specialized serializer.
> > > > >
> > > > > So I would like to know from the community what do think of
> changing
> > > the
> > > > > way the optimized marshaller works or introducing the support of
> yet
> > > > > another marshaller based on a kryo like technology?
> > > > >
> > > > >
> > > > > Thanks in advance,
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > >
> > > > > Romain.
> > > > >
> > > > >
> > > > > PS: I'm ready to help in the both cases.
> > > > > "Misys" is the trade name of the Misys group of companies. This
> email
> > > and
> > > > > any attachments have been scanned for known viruses using multiple
> > > > > scanners. This email message is intended for the named recipient
> only.
> > > It
> > > > > may be privileged and/or confidential. If you are not the named
> > > recipient
> > > > > of this email please notify us immediately and do not copy it or
> use
> > > it for
> > > > > any purpose, nor disclose its contents to any other person. This
> email
> > > does
> > > > > not constitute the commencement of legal relations between you and
> > > Misys.
> > > > > Please refer to the executed contract between you and the relevant
> > > member
> > > > > of the Misys group for the identity of the contracting party with
> > > which you
> > > > > are dealing.
> > > > >
> > >
>
>
RE: OSGi migration may require a special marshaller
Posted by Andrey Kornev <an...@hotmail.com>.
Hey Raul,
What kind of use case do you have in mind when you suggest that a classdef may be located in different bundles on different machines? What would one have to do to end up in such a predicament? It feels more like a configuration/deployment issue rather than a desirable feature.
A more realistic assumption I believe is that the nodes will have (mostly) the same set of bundles deployed. (We even enforce it at runtime by leveraging Ignite's PluginProvider.validateNewNode() method: we do not allow new nodes join the cluster if their OSGi environment is inconsistent.)
The "bundle name" in my original email really meant "bundle symbolic name + bundle version". Sorry for not being clear.
I'm not sure I understand why TCCL is even required for serialization?
Finally, I didn't quite grok your thinking about the custom manifest header. Could you please elaborate? What type of information do you propose to store in the header? How will the serializer be able to provide that info, and how will the deserializer use the info to load the classes? Also, who is responsible for making sure that the custom header includes all classes to be serialized? And how/when/by whom would such a header be generated?
Regards
Andrey
> Date: Mon, 2 Nov 2015 23:20:41 +0000
> Subject: RE: OSGi migration may require a special marshaller
> From: raul@evosent.com
> To: dev@ignite.apache.org
>
> Hi Andrey,
>
> Thanks for the participation in this topic.
>
> I don't like the solution to incorporate the bundle symbolic name in the
> serialised form. Nothing guarantees that the classdef will be located under
> the same bundle in both source and target machines. We also have to take
> into account that OSGi allows for multiple versions of the same
> bundle/packages to co-exist in the same container. So it becomes more
> complex.
>
> Using the TCCL should work when serialising, but it'll probably be of no
> use when deserialising on the other end.
>
> I need to enhance Ignite to:
>
> 1. Use the TCCL when marshalling on one end.
> 2. Incorporate the package version of the class in the serialised form if
> Ignite is running in an OSGi environment.
> 3. On the receiver end, discover cache entities / DTOs in all bundles
> through a custom OSGi manifest header or the like, as I explained before.
> Package version must be taken into account.
>
> What do you think?
>
> Raúl.
> On 2 Nov 2015 17:25, "Andrey Kornev" <an...@hotmail.com> wrote:
>
> > Raul,
> >
> > The fundamental hurdle we need to jump over to make Ignite OSGi-enabled is
> > the marshalling. More specifically the issue is with deserialization of the
> > classes that are provided by the bundles other than the Ignite bundle
> > itself.
> >
> > When the Ignite transport layer receives a message it needs to figure out
> > how to deserialize the bytes and for that it needs to know the bundle that
> > provides the class to be deserailized. At this point TCCL is of no use. To
> > make things more complex, the class may contain other classes that come
> > from other bundles, and so on recursively. This means that each object in
> > the hierarchy must be serialized with its bundle name (or bundle id), so
> > that the deserializer will then be able to correctly resolve the class
> > while traversing the object hierarchy during deserialization.
> >
> > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to plug
> > in a custom class resolver. Romain's solution was to use Kryo that does
> > provide a way to customize class resolution. It has solved the problem of
> > capturing the bundle info and he was able to successfully run Ignite as a
> > bundle in an OSGi container (after some repackaging and inclusion of the
> > manifest). But Kryo-based marshalling introduced a lot of complexity to the
> > code and incorrect use of Kryo's numerous serializers caused some weird
> > hard-to-debug issues in the Ignite core (like duplicate cache entries due
> > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > figure!). Overall the Kryo-based solution is brittle and hard to maintain.
> >
> > I feel the correct solution to OSGi problem would be to
> > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > original one) to be used in OSGi environment.
> >
> > Regards
> > Andrey
> >
> > > From: raulk@apache.org
> > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: dev@ignite.apache.org
> > >
> > > Hi Romain,
> > >
> > > I'm working on the OSGi compatibility of Ignite. I appreciate your input.
> > >
> > > I'm thinking about the situation you describe and I wonder if you're
> > > exporting Ignite as an OSGi service which is then consumed from other
> > > bundles. Under this situation, it would be quite easy to reproduce the
> > > behaviour you describe if Ignite is not resolving classes via the TCCL.
> > > Need to dig deeper into that.
> > >
> > > Off the top of my head, there are two alternatives to solve it:
> > >
> > > 1. Use the TCCL for marshalling/unmarshalling (if not already used) – we
> > > gotta be wary of possible regressions.
> > > 2. Create a special OSGi header Ignite-Export-Package so that bundles
> > > containing DTOs can expose packages to Ignite's marshallers.
> > >
> > > Regards,
> > >
> > > *Raúl Kripalani*
> > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> > > Messaging Engineer
> > > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > > http://blog.raulkr.net | twitter: @raulvk
> > >
> > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <ro...@misys.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm really interested in this issue:
> > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to
> > make
> > > > it work in our osgi environment. The main issue for us now is the
> > > > serialization. I think it you will have to rework the
> > OptimizedMarshaller
> > > > or any other marshaller that works with object that come from outside
> > your
> > > > class space.
> > > >
> > > > We have try kryo that works. Kryo provide an extension point in order
> > to
> > > > resolve the classes:
> > > >
> > https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > . With this extension we are able to solve the problem of external
> > classes.
> > > > The only issue with kryo is that some classes need a certain care in
> > the
> > > > serialization process and therefore a specialized serializer.
> > > >
> > > > So I would like to know from the community what do think of changing
> > the
> > > > way the optimized marshaller works or introducing the support of yet
> > > > another marshaller based on a kryo like technology?
> > > >
> > > >
> > > > Thanks in advance,
> > > >
> > > >
> > > > Best regards,
> > > >
> > > >
> > > > Romain.
> > > >
> > > >
> > > > PS: I'm ready to help in the both cases.
> > > > "Misys" is the trade name of the Misys group of companies. This email
> > and
> > > > any attachments have been scanned for known viruses using multiple
> > > > scanners. This email message is intended for the named recipient only.
> > It
> > > > may be privileged and/or confidential. If you are not the named
> > recipient
> > > > of this email please notify us immediately and do not copy it or use
> > it for
> > > > any purpose, nor disclose its contents to any other person. This email
> > does
> > > > not constitute the commencement of legal relations between you and
> > Misys.
> > > > Please refer to the executed contract between you and the relevant
> > member
> > > > of the Misys group for the identity of the contracting party with
> > which you
> > > > are dealing.
> > > >
> >
Re: OSGi migration may require a special marshaller
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Raul,
Why do we need TCCL for serializing? We already have the Object we want to
serialize, so to get its class loader we can call
myObject.getClass().getClassLoader(), no?
D.
On Mon, Nov 2, 2015 at 3:20 PM, Raul Kripalani <ra...@evosent.com> wrote:
> Hi Andrey,
>
> Thanks for the participation in this topic.
>
> I don't like the solution to incorporate the bundle symbolic name in the
> serialised form. Nothing guarantees that the classdef will be located under
> the same bundle in both source and target machines. We also have to take
> into account that OSGi allows for multiple versions of the same
> bundle/packages to co-exist in the same container. So it becomes more
> complex.
>
> Using the TCCL should work when serialising, but it'll probably be of no
> use when deserialising on the other end.
>
> I need to enhance Ignite to:
>
> 1. Use the TCCL when marshalling on one end.
> 2. Incorporate the package version of the class in the serialised form if
> Ignite is running in an OSGi environment.
> 3. On the receiver end, discover cache entities / DTOs in all bundles
> through a custom OSGi manifest header or the like, as I explained before.
> Package version must be taken into account.
>
> What do you think?
>
> Raúl.
> On 2 Nov 2015 17:25, "Andrey Kornev" <an...@hotmail.com> wrote:
>
> > Raul,
> >
> > The fundamental hurdle we need to jump over to make Ignite OSGi-enabled
> is
> > the marshalling. More specifically the issue is with deserialization of
> the
> > classes that are provided by the bundles other than the Ignite bundle
> > itself.
> >
> > When the Ignite transport layer receives a message it needs to figure out
> > how to deserialize the bytes and for that it needs to know the bundle
> that
> > provides the class to be deserailized. At this point TCCL is of no use.
> To
> > make things more complex, the class may contain other classes that come
> > from other bundles, and so on recursively. This means that each object in
> > the hierarchy must be serialized with its bundle name (or bundle id), so
> > that the deserializer will then be able to correctly resolve the class
> > while traversing the object hierarchy during deserialization.
> >
> > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> plug
> > in a custom class resolver. Romain's solution was to use Kryo that does
> > provide a way to customize class resolution. It has solved the problem of
> > capturing the bundle info and he was able to successfully run Ignite as a
> > bundle in an OSGi container (after some repackaging and inclusion of the
> > manifest). But Kryo-based marshalling introduced a lot of complexity to
> the
> > code and incorrect use of Kryo's numerous serializers caused some weird
> > hard-to-debug issues in the Ignite core (like duplicate cache entries due
> > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > figure!). Overall the Kryo-based solution is brittle and hard to
> maintain.
> >
> > I feel the correct solution to OSGi problem would be to
> > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > original one) to be used in OSGi environment.
> >
> > Regards
> > Andrey
> >
> > > From: raulk@apache.org
> > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: dev@ignite.apache.org
> > >
> > > Hi Romain,
> > >
> > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> input.
> > >
> > > I'm thinking about the situation you describe and I wonder if you're
> > > exporting Ignite as an OSGi service which is then consumed from other
> > > bundles. Under this situation, it would be quite easy to reproduce the
> > > behaviour you describe if Ignite is not resolving classes via the TCCL.
> > > Need to dig deeper into that.
> > >
> > > Off the top of my head, there are two alternatives to solve it:
> > >
> > > 1. Use the TCCL for marshalling/unmarshalling (if not already used) –
> we
> > > gotta be wary of possible regressions.
> > > 2. Create a special OSGi header Ignite-Export-Package so that bundles
> > > containing DTOs can expose packages to Ignite's marshallers.
> > >
> > > Regards,
> > >
> > > *Raúl Kripalani*
> > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> and
> > > Messaging Engineer
> > > http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> > > http://blog.raulkr.net | twitter: @raulvk
> > >
> > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> romain.gilles@misys.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm really interested in this issue:
> > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to
> > make
> > > > it work in our osgi environment. The main issue for us now is the
> > > > serialization. I think it you will have to rework the
> > OptimizedMarshaller
> > > > or any other marshaller that works with object that come from outside
> > your
> > > > class space.
> > > >
> > > > We have try kryo that works. Kryo provide an extension point in order
> > to
> > > > resolve the classes:
> > > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > . With this extension we are able to solve the problem of external
> > classes.
> > > > The only issue with kryo is that some classes need a certain care in
> > the
> > > > serialization process and therefore a specialized serializer.
> > > >
> > > > So I would like to know from the community what do think of changing
> > the
> > > > way the optimized marshaller works or introducing the support of yet
> > > > another marshaller based on a kryo like technology?
> > > >
> > > >
> > > > Thanks in advance,
> > > >
> > > >
> > > > Best regards,
> > > >
> > > >
> > > > Romain.
> > > >
> > > >
> > > > PS: I'm ready to help in the both cases.
> > > > "Misys" is the trade name of the Misys group of companies. This email
> > and
> > > > any attachments have been scanned for known viruses using multiple
> > > > scanners. This email message is intended for the named recipient
> only.
> > It
> > > > may be privileged and/or confidential. If you are not the named
> > recipient
> > > > of this email please notify us immediately and do not copy it or use
> > it for
> > > > any purpose, nor disclose its contents to any other person. This
> email
> > does
> > > > not constitute the commencement of legal relations between you and
> > Misys.
> > > > Please refer to the executed contract between you and the relevant
> > member
> > > > of the Misys group for the identity of the contracting party with
> > which you
> > > > are dealing.
> > > >
> >
>
RE: OSGi migration may require a special marshaller
Posted by Raul Kripalani <ra...@evosent.com>.
Hi Andrey,
Thanks for the participation in this topic.
I don't like the solution to incorporate the bundle symbolic name in the
serialised form. Nothing guarantees that the classdef will be located under
the same bundle in both source and target machines. We also have to take
into account that OSGi allows for multiple versions of the same
bundle/packages to co-exist in the same container. So it becomes more
complex.
Using the TCCL should work when serialising, but it'll probably be of no
use when deserialising on the other end.
I need to enhance Ignite to:
1. Use the TCCL when marshalling on one end.
2. Incorporate the package version of the class in the serialised form if
Ignite is running in an OSGi environment.
3. On the receiver end, discover cache entities / DTOs in all bundles
through a custom OSGi manifest header or the like, as I explained before.
Package version must be taken into account.
What do you think?
Raúl.
On 2 Nov 2015 17:25, "Andrey Kornev" <an...@hotmail.com> wrote:
> Raul,
>
> The fundamental hurdle we need to jump over to make Ignite OSGi-enabled is
> the marshalling. More specifically the issue is with deserialization of the
> classes that are provided by the bundles other than the Ignite bundle
> itself.
>
> When the Ignite transport layer receives a message it needs to figure out
> how to deserialize the bytes and for that it needs to know the bundle that
> provides the class to be deserailized. At this point TCCL is of no use. To
> make things more complex, the class may contain other classes that come
> from other bundles, and so on recursively. This means that each object in
> the hierarchy must be serialized with its bundle name (or bundle id), so
> that the deserializer will then be able to correctly resolve the class
> while traversing the object hierarchy during deserialization.
>
> Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to plug
> in a custom class resolver. Romain's solution was to use Kryo that does
> provide a way to customize class resolution. It has solved the problem of
> capturing the bundle info and he was able to successfully run Ignite as a
> bundle in an OSGi container (after some repackaging and inclusion of the
> manifest). But Kryo-based marshalling introduced a lot of complexity to the
> code and incorrect use of Kryo's numerous serializers caused some weird
> hard-to-debug issues in the Ignite core (like duplicate cache entries due
> to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> figure!). Overall the Kryo-based solution is brittle and hard to maintain.
>
> I feel the correct solution to OSGi problem would be to
> 1) enhance the OptimizedMarshaller to allow custom class resolution.
> 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> original one) to be used in OSGi environment.
>
> Regards
> Andrey
>
> > From: raulk@apache.org
> > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > Subject: Re: OSGi migration may require a special marshaller
> > To: dev@ignite.apache.org
> >
> > Hi Romain,
> >
> > I'm working on the OSGi compatibility of Ignite. I appreciate your input.
> >
> > I'm thinking about the situation you describe and I wonder if you're
> > exporting Ignite as an OSGi service which is then consumed from other
> > bundles. Under this situation, it would be quite easy to reproduce the
> > behaviour you describe if Ignite is not resolving classes via the TCCL.
> > Need to dig deeper into that.
> >
> > Off the top of my head, there are two alternatives to solve it:
> >
> > 1. Use the TCCL for marshalling/unmarshalling (if not already used) – we
> > gotta be wary of possible regressions.
> > 2. Create a special OSGi header Ignite-Export-Package so that bundles
> > containing DTOs can expose packages to Ignite's marshallers.
> >
> > Regards,
> >
> > *Raúl Kripalani*
> > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> > Messaging Engineer
> > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > http://blog.raulkr.net | twitter: @raulvk
> >
> > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <ro...@misys.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I'm really interested in this issue:
> > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to
> make
> > > it work in our osgi environment. The main issue for us now is the
> > > serialization. I think it you will have to rework the
> OptimizedMarshaller
> > > or any other marshaller that works with object that come from outside
> your
> > > class space.
> > >
> > > We have try kryo that works. Kryo provide an extension point in order
> to
> > > resolve the classes:
> > >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > . With this extension we are able to solve the problem of external
> classes.
> > > The only issue with kryo is that some classes need a certain care in
> the
> > > serialization process and therefore a specialized serializer.
> > >
> > > So I would like to know from the community what do think of changing
> the
> > > way the optimized marshaller works or introducing the support of yet
> > > another marshaller based on a kryo like technology?
> > >
> > >
> > > Thanks in advance,
> > >
> > >
> > > Best regards,
> > >
> > >
> > > Romain.
> > >
> > >
> > > PS: I'm ready to help in the both cases.
> > > "Misys" is the trade name of the Misys group of companies. This email
> and
> > > any attachments have been scanned for known viruses using multiple
> > > scanners. This email message is intended for the named recipient only.
> It
> > > may be privileged and/or confidential. If you are not the named
> recipient
> > > of this email please notify us immediately and do not copy it or use
> it for
> > > any purpose, nor disclose its contents to any other person. This email
> does
> > > not constitute the commencement of legal relations between you and
> Misys.
> > > Please refer to the executed contract between you and the relevant
> member
> > > of the Misys group for the identity of the contracting party with
> which you
> > > are dealing.
> > >
>
RE: OSGi migration may require a special marshaller
Posted by Andrey Kornev <an...@hotmail.com>.
Raul,
The fundamental hurdle we need to jump over to make Ignite OSGi-enabled is the marshalling. More specifically the issue is with deserialization of the classes that are provided by the bundles other than the Ignite bundle itself.
When the Ignite transport layer receives a message it needs to figure out how to deserialize the bytes and for that it needs to know the bundle that provides the class to be deserailized. At this point TCCL is of no use. To make things more complex, the class may contain other classes that come from other bundles, and so on recursively. This means that each object in the hierarchy must be serialized with its bundle name (or bundle id), so that the deserializer will then be able to correctly resolve the class while traversing the object hierarchy during deserialization.
Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to plug in a custom class resolver. Romain's solution was to use Kryo that does provide a way to customize class resolution. It has solved the problem of capturing the bundle info and he was able to successfully run Ignite as a bundle in an OSGi container (after some repackaging and inclusion of the manifest). But Kryo-based marshalling introduced a lot of complexity to the code and incorrect use of Kryo's numerous serializers caused some weird hard-to-debug issues in the Ignite core (like duplicate cache entries due to incorrect marshalling of the GridDhtPArtitonFullMap class -- go figure!). Overall the Kryo-based solution is brittle and hard to maintain.
I feel the correct solution to OSGi problem would be to
1) enhance the OptimizedMarshaller to allow custom class resolution.
2) provide an OSGi-enabled OptimizedMarshaller (in addition to the original one) to be used in OSGi environment.
Regards
Andrey
> From: raulk@apache.org
> Date: Mon, 2 Nov 2015 12:41:47 +0000
> Subject: Re: OSGi migration may require a special marshaller
> To: dev@ignite.apache.org
>
> Hi Romain,
>
> I'm working on the OSGi compatibility of Ignite. I appreciate your input.
>
> I'm thinking about the situation you describe and I wonder if you're
> exporting Ignite as an OSGi service which is then consumed from other
> bundles. Under this situation, it would be quite easy to reproduce the
> behaviour you describe if Ignite is not resolving classes via the TCCL.
> Need to dig deeper into that.
>
> Off the top of my head, there are two alternatives to solve it:
>
> 1. Use the TCCL for marshalling/unmarshalling (if not already used) – we
> gotta be wary of possible regressions.
> 2. Create a special OSGi header Ignite-Export-Package so that bundles
> containing DTOs can expose packages to Ignite's marshallers.
>
> Regards,
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>
> On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <ro...@misys.com>
> wrote:
>
> > Hi all,
> >
> > I'm really interested in this issue:
> > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to make
> > it work in our osgi environment. The main issue for us now is the
> > serialization. I think it you will have to rework the OptimizedMarshaller
> > or any other marshaller that works with object that come from outside your
> > class space.
> >
> > We have try kryo that works. Kryo provide an extension point in order to
> > resolve the classes:
> > https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > . With this extension we are able to solve the problem of external classes.
> > The only issue with kryo is that some classes need a certain care in the
> > serialization process and therefore a specialized serializer.
> >
> > So I would like to know from the community what do think of changing the
> > way the optimized marshaller works or introducing the support of yet
> > another marshaller based on a kryo like technology?
> >
> >
> > Thanks in advance,
> >
> >
> > Best regards,
> >
> >
> > Romain.
> >
> >
> > PS: I'm ready to help in the both cases.
> > "Misys" is the trade name of the Misys group of companies. This email and
> > any attachments have been scanned for known viruses using multiple
> > scanners. This email message is intended for the named recipient only. It
> > may be privileged and/or confidential. If you are not the named recipient
> > of this email please notify us immediately and do not copy it or use it for
> > any purpose, nor disclose its contents to any other person. This email does
> > not constitute the commencement of legal relations between you and Misys.
> > Please refer to the executed contract between you and the relevant member
> > of the Misys group for the identity of the contracting party with which you
> > are dealing.
> >
Re: OSGi migration may require a special marshaller
Posted by Raul Kripalani <ra...@apache.org>.
Hi Romain,
I'm working on the OSGi compatibility of Ignite. I appreciate your input.
I'm thinking about the situation you describe and I wonder if you're
exporting Ignite as an OSGi service which is then consumed from other
bundles. Under this situation, it would be quite easy to reproduce the
behaviour you describe if Ignite is not resolving classes via the TCCL.
Need to dig deeper into that.
Off the top of my head, there are two alternatives to solve it:
1. Use the TCCL for marshalling/unmarshalling (if not already used) – we
gotta be wary of possible regressions.
2. Create a special OSGi header Ignite-Export-Package so that bundles
containing DTOs can expose packages to Ignite's marshallers.
Regards,
*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk
On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <ro...@misys.com>
wrote:
> Hi all,
>
> I'm really interested in this issue:
> https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff to make
> it work in our osgi environment. The main issue for us now is the
> serialization. I think it you will have to rework the OptimizedMarshaller
> or any other marshaller that works with object that come from outside your
> class space.
>
> We have try kryo that works. Kryo provide an extension point in order to
> resolve the classes:
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> . With this extension we are able to solve the problem of external classes.
> The only issue with kryo is that some classes need a certain care in the
> serialization process and therefore a specialized serializer.
>
> So I would like to know from the community what do think of changing the
> way the optimized marshaller works or introducing the support of yet
> another marshaller based on a kryo like technology?
>
>
> Thanks in advance,
>
>
> Best regards,
>
>
> Romain.
>
>
> PS: I'm ready to help in the both cases.
> "Misys" is the trade name of the Misys group of companies. This email and
> any attachments have been scanned for known viruses using multiple
> scanners. This email message is intended for the named recipient only. It
> may be privileged and/or confidential. If you are not the named recipient
> of this email please notify us immediately and do not copy it or use it for
> any purpose, nor disclose its contents to any other person. This email does
> not constitute the commencement of legal relations between you and Misys.
> Please refer to the executed contract between you and the relevant member
> of the Misys group for the identity of the contracting party with which you
> are dealing.
>