You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by Stephen Mallette <sp...@gmail.com> on 2017/06/20 12:25:33 UTC
[DISCUSS] Gryo 3.0
I'd really like to get our IO situation really solid in 3.3.0 and while we
talk about GraphSON a lot I think Gryo still has an important place in our
serialization story. Gryo covers an extended set of classes that goes
beyond what we cover in GraphSON. I don't know if all those registered
classes matter though - one would think that we just need parity with
GraphSON defined classes.
https://github.com/apache/tinkerpop/blob/9f13e69853c03a2fde77efd40c6526b91a4b346a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoVersion.java#L192
I think that the reason this area of code has grown so much is from the
basis of another issue with gryo which is that we don't always have a nice
separation between implementations and serialized classes. In other words,
if we have some complex class like DefaultTraversalMetrics, we just
register all the classes that DefaultTraversalMetrics has as fields (and
all the classes those classes use) until the serialization works. While
this is really easy and, in a sense, convenient, it creates some problems:
1. We get this big long registration list which includes types we don't
support in GraphSON (which typically aren't tested)
2. We have now bound the serialization of the complex class to its
structure. If there is a problem with DefaultTraversalMetrics in the future
and we have to change a field we essentially break all compatibility within
gryo. (i'm facing this issue now actually).
I would really like Gryo 3.0 to do a better job in this area, allowing for
better forward and backward compatibility with a more limited class
registration list that is more in line with the types defined for GraphSON
support. I think we can get better forward/backward compatibility if (1)
there is better separation between the object being serialized and the
object used in application logic (think of the "detached" classes) and (2)
gryo serializers for 3.0 have greater intelligence around internal
compatibility handling.
Anyway, I was mostly writing this as notes to myself in preparation for
3.3.0, but I'd be happy to hear any thoughts anyone might have on this.
Re: [DISCUSS] Gryo 3.0
Posted by Stephen Mallette <sp...@gmail.com>.
Marko digging into TINKERPOP-1592 (his comments on the "[DISCUSS] Remaining
Items for 3.3.0") has made me basically realize there isn't much value to
doing Gryo 3.0 at this point. More reasoning over on that thread.
On Tue, Jun 27, 2017 at 9:25 AM, Stephen Mallette <sp...@gmail.com>
wrote:
> We currently have a wide range of serialization options in Gryo around
> different types of "detached" Vertex/Edge instances which typically just
> used the default Kryo FieldSerializer. As mentioned earlier in this thread,
> I've wanted to change that pattern a bit for Gryo 3.0. I think that we can
> have one serializer that will efficiently serialize each type of "detached"
> Vertex/Edge that we have. When done in conjunction with TINKERPOP-1592 I
> think we will have a good solid design in place for 3.3.0.
>
> You can take a look at what's in store for the Gryo 3.0 Element
> serializers so far here:
>
> https://github.com/apache/tinkerpop/blob/TINKERPOP-1698/
> gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/
> GryoSerializersV3d0.java
>
> Hopefully this branch can form into a PR within the coming days.
>
>
>
> On Mon, Jun 26, 2017 at 6:22 PM, Stephen Mallette <sp...@gmail.com>
> wrote:
>
>> Now that we have gremlin-io-test for 3.3.0 which tests different versions
>> of gryo/graphson over different versions of tinkerpop (meaning the current
>> version you build with ensure that it can still read/write the old formats)
>> do we need to do this:
>>
>> https://issues.apache.org/jira/browse/TINKERPOP-1364
>>
>> I tend to think the answer is "no" at this point. I think that for the
>> distributions we just need to have Gryo and GraphSON 3.0 and that's it (and
>> obviously the existing GraphML files). If we do that we should get rid of a
>> dozen files or so. Anyway, unless there are objections in the next 72 hours
>> I will assume lazy consensus and move forward in that direction.
>>
>>
>>
>> On Mon, Jun 26, 2017 at 10:24 AM, Stephen Mallette <sp...@gmail.com>
>> wrote:
>>
>>> Just wanted to note that Gryo 3.0 work is happening here in case anyone
>>> wants to follow along:
>>>
>>> https://issues.apache.org/jira/browse/TINKERPOP-1698
>>>
>>> I have a branch setup. I think I'm going to clean up Gremlin Server
>>> configuration files for 3.3.0 and remove a bunch of the old serializers.
>>> The config is starting to look really confusing. I think it would be best
>>> to just have Gryo and GraphSON 3.0 configured in the packaged sample files.
>>>
>>>
>>> On Tue, Jun 20, 2017 at 8:25 AM, Stephen Mallette <sp...@gmail.com>
>>> wrote:
>>>
>>>> I'd really like to get our IO situation really solid in 3.3.0 and while
>>>> we talk about GraphSON a lot I think Gryo still has an important place in
>>>> our serialization story. Gryo covers an extended set of classes that goes
>>>> beyond what we cover in GraphSON. I don't know if all those registered
>>>> classes matter though - one would think that we just need parity with
>>>> GraphSON defined classes.
>>>>
>>>> https://github.com/apache/tinkerpop/blob/9f13e69853c03a2fde7
>>>> 7efd40c6526b91a4b346a/gremlin-core/src/main/java/org/apache/
>>>> tinkerpop/gremlin/structure/io/gryo/GryoVersion.java#L192
>>>>
>>>> I think that the reason this area of code has grown so much is from the
>>>> basis of another issue with gryo which is that we don't always have a nice
>>>> separation between implementations and serialized classes. In other words,
>>>> if we have some complex class like DefaultTraversalMetrics, we just
>>>> register all the classes that DefaultTraversalMetrics has as fields (and
>>>> all the classes those classes use) until the serialization works. While
>>>> this is really easy and, in a sense, convenient, it creates some problems:
>>>>
>>>> 1. We get this big long registration list which includes types we don't
>>>> support in GraphSON (which typically aren't tested)
>>>> 2. We have now bound the serialization of the complex class to its
>>>> structure. If there is a problem with DefaultTraversalMetrics in the future
>>>> and we have to change a field we essentially break all compatibility within
>>>> gryo. (i'm facing this issue now actually).
>>>>
>>>> I would really like Gryo 3.0 to do a better job in this area, allowing
>>>> for better forward and backward compatibility with a more limited class
>>>> registration list that is more in line with the types defined for GraphSON
>>>> support. I think we can get better forward/backward compatibility if (1)
>>>> there is better separation between the object being serialized and the
>>>> object used in application logic (think of the "detached" classes) and (2)
>>>> gryo serializers for 3.0 have greater intelligence around internal
>>>> compatibility handling.
>>>>
>>>> Anyway, I was mostly writing this as notes to myself in preparation for
>>>> 3.3.0, but I'd be happy to hear any thoughts anyone might have on this.
>>>>
>>>
>>>
>>
>
Re: [DISCUSS] Gryo 3.0
Posted by Stephen Mallette <sp...@gmail.com>.
We currently have a wide range of serialization options in Gryo around
different types of "detached" Vertex/Edge instances which typically just
used the default Kryo FieldSerializer. As mentioned earlier in this thread,
I've wanted to change that pattern a bit for Gryo 3.0. I think that we can
have one serializer that will efficiently serialize each type of "detached"
Vertex/Edge that we have. When done in conjunction with TINKERPOP-1592 I
think we will have a good solid design in place for 3.3.0.
You can take a look at what's in store for the Gryo 3.0 Element serializers
so far here:
https://github.com/apache/tinkerpop/blob/TINKERPOP-1698/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/gryo/GryoSerializersV3d0.java
Hopefully this branch can form into a PR within the coming days.
On Mon, Jun 26, 2017 at 6:22 PM, Stephen Mallette <sp...@gmail.com>
wrote:
> Now that we have gremlin-io-test for 3.3.0 which tests different versions
> of gryo/graphson over different versions of tinkerpop (meaning the current
> version you build with ensure that it can still read/write the old formats)
> do we need to do this:
>
> https://issues.apache.org/jira/browse/TINKERPOP-1364
>
> I tend to think the answer is "no" at this point. I think that for the
> distributions we just need to have Gryo and GraphSON 3.0 and that's it (and
> obviously the existing GraphML files). If we do that we should get rid of a
> dozen files or so. Anyway, unless there are objections in the next 72 hours
> I will assume lazy consensus and move forward in that direction.
>
>
>
> On Mon, Jun 26, 2017 at 10:24 AM, Stephen Mallette <sp...@gmail.com>
> wrote:
>
>> Just wanted to note that Gryo 3.0 work is happening here in case anyone
>> wants to follow along:
>>
>> https://issues.apache.org/jira/browse/TINKERPOP-1698
>>
>> I have a branch setup. I think I'm going to clean up Gremlin Server
>> configuration files for 3.3.0 and remove a bunch of the old serializers.
>> The config is starting to look really confusing. I think it would be best
>> to just have Gryo and GraphSON 3.0 configured in the packaged sample files.
>>
>>
>> On Tue, Jun 20, 2017 at 8:25 AM, Stephen Mallette <sp...@gmail.com>
>> wrote:
>>
>>> I'd really like to get our IO situation really solid in 3.3.0 and while
>>> we talk about GraphSON a lot I think Gryo still has an important place in
>>> our serialization story. Gryo covers an extended set of classes that goes
>>> beyond what we cover in GraphSON. I don't know if all those registered
>>> classes matter though - one would think that we just need parity with
>>> GraphSON defined classes.
>>>
>>> https://github.com/apache/tinkerpop/blob/9f13e69853c03a2fde7
>>> 7efd40c6526b91a4b346a/gremlin-core/src/main/java/org/apache/
>>> tinkerpop/gremlin/structure/io/gryo/GryoVersion.java#L192
>>>
>>> I think that the reason this area of code has grown so much is from the
>>> basis of another issue with gryo which is that we don't always have a nice
>>> separation between implementations and serialized classes. In other words,
>>> if we have some complex class like DefaultTraversalMetrics, we just
>>> register all the classes that DefaultTraversalMetrics has as fields (and
>>> all the classes those classes use) until the serialization works. While
>>> this is really easy and, in a sense, convenient, it creates some problems:
>>>
>>> 1. We get this big long registration list which includes types we don't
>>> support in GraphSON (which typically aren't tested)
>>> 2. We have now bound the serialization of the complex class to its
>>> structure. If there is a problem with DefaultTraversalMetrics in the future
>>> and we have to change a field we essentially break all compatibility within
>>> gryo. (i'm facing this issue now actually).
>>>
>>> I would really like Gryo 3.0 to do a better job in this area, allowing
>>> for better forward and backward compatibility with a more limited class
>>> registration list that is more in line with the types defined for GraphSON
>>> support. I think we can get better forward/backward compatibility if (1)
>>> there is better separation between the object being serialized and the
>>> object used in application logic (think of the "detached" classes) and (2)
>>> gryo serializers for 3.0 have greater intelligence around internal
>>> compatibility handling.
>>>
>>> Anyway, I was mostly writing this as notes to myself in preparation for
>>> 3.3.0, but I'd be happy to hear any thoughts anyone might have on this.
>>>
>>
>>
>
Re: [DISCUSS] Gryo 3.0
Posted by Stephen Mallette <sp...@gmail.com>.
Now that we have gremlin-io-test for 3.3.0 which tests different versions
of gryo/graphson over different versions of tinkerpop (meaning the current
version you build with ensure that it can still read/write the old formats)
do we need to do this:
https://issues.apache.org/jira/browse/TINKERPOP-1364
I tend to think the answer is "no" at this point. I think that for the
distributions we just need to have Gryo and GraphSON 3.0 and that's it (and
obviously the existing GraphML files). If we do that we should get rid of a
dozen files or so. Anyway, unless there are objections in the next 72 hours
I will assume lazy consensus and move forward in that direction.
On Mon, Jun 26, 2017 at 10:24 AM, Stephen Mallette <sp...@gmail.com>
wrote:
> Just wanted to note that Gryo 3.0 work is happening here in case anyone
> wants to follow along:
>
> https://issues.apache.org/jira/browse/TINKERPOP-1698
>
> I have a branch setup. I think I'm going to clean up Gremlin Server
> configuration files for 3.3.0 and remove a bunch of the old serializers.
> The config is starting to look really confusing. I think it would be best
> to just have Gryo and GraphSON 3.0 configured in the packaged sample files.
>
>
> On Tue, Jun 20, 2017 at 8:25 AM, Stephen Mallette <sp...@gmail.com>
> wrote:
>
>> I'd really like to get our IO situation really solid in 3.3.0 and while
>> we talk about GraphSON a lot I think Gryo still has an important place in
>> our serialization story. Gryo covers an extended set of classes that goes
>> beyond what we cover in GraphSON. I don't know if all those registered
>> classes matter though - one would think that we just need parity with
>> GraphSON defined classes.
>>
>> https://github.com/apache/tinkerpop/blob/9f13e69853c03a2fde7
>> 7efd40c6526b91a4b346a/gremlin-core/src/main/java/org/apache/
>> tinkerpop/gremlin/structure/io/gryo/GryoVersion.java#L192
>>
>> I think that the reason this area of code has grown so much is from the
>> basis of another issue with gryo which is that we don't always have a nice
>> separation between implementations and serialized classes. In other words,
>> if we have some complex class like DefaultTraversalMetrics, we just
>> register all the classes that DefaultTraversalMetrics has as fields (and
>> all the classes those classes use) until the serialization works. While
>> this is really easy and, in a sense, convenient, it creates some problems:
>>
>> 1. We get this big long registration list which includes types we don't
>> support in GraphSON (which typically aren't tested)
>> 2. We have now bound the serialization of the complex class to its
>> structure. If there is a problem with DefaultTraversalMetrics in the future
>> and we have to change a field we essentially break all compatibility within
>> gryo. (i'm facing this issue now actually).
>>
>> I would really like Gryo 3.0 to do a better job in this area, allowing
>> for better forward and backward compatibility with a more limited class
>> registration list that is more in line with the types defined for GraphSON
>> support. I think we can get better forward/backward compatibility if (1)
>> there is better separation between the object being serialized and the
>> object used in application logic (think of the "detached" classes) and (2)
>> gryo serializers for 3.0 have greater intelligence around internal
>> compatibility handling.
>>
>> Anyway, I was mostly writing this as notes to myself in preparation for
>> 3.3.0, but I'd be happy to hear any thoughts anyone might have on this.
>>
>
>
Re: [DISCUSS] Gryo 3.0
Posted by Stephen Mallette <sp...@gmail.com>.
Just wanted to note that Gryo 3.0 work is happening here in case anyone
wants to follow along:
https://issues.apache.org/jira/browse/TINKERPOP-1698
I have a branch setup. I think I'm going to clean up Gremlin Server
configuration files for 3.3.0 and remove a bunch of the old serializers.
The config is starting to look really confusing. I think it would be best
to just have Gryo and GraphSON 3.0 configured in the packaged sample files.
On Tue, Jun 20, 2017 at 8:25 AM, Stephen Mallette <sp...@gmail.com>
wrote:
> I'd really like to get our IO situation really solid in 3.3.0 and while we
> talk about GraphSON a lot I think Gryo still has an important place in our
> serialization story. Gryo covers an extended set of classes that goes
> beyond what we cover in GraphSON. I don't know if all those registered
> classes matter though - one would think that we just need parity with
> GraphSON defined classes.
>
> https://github.com/apache/tinkerpop/blob/9f13e69853c03a2fde77efd40c6526
> b91a4b346a/gremlin-core/src/main/java/org/apache/
> tinkerpop/gremlin/structure/io/gryo/GryoVersion.java#L192
>
> I think that the reason this area of code has grown so much is from the
> basis of another issue with gryo which is that we don't always have a nice
> separation between implementations and serialized classes. In other words,
> if we have some complex class like DefaultTraversalMetrics, we just
> register all the classes that DefaultTraversalMetrics has as fields (and
> all the classes those classes use) until the serialization works. While
> this is really easy and, in a sense, convenient, it creates some problems:
>
> 1. We get this big long registration list which includes types we don't
> support in GraphSON (which typically aren't tested)
> 2. We have now bound the serialization of the complex class to its
> structure. If there is a problem with DefaultTraversalMetrics in the future
> and we have to change a field we essentially break all compatibility within
> gryo. (i'm facing this issue now actually).
>
> I would really like Gryo 3.0 to do a better job in this area, allowing for
> better forward and backward compatibility with a more limited class
> registration list that is more in line with the types defined for GraphSON
> support. I think we can get better forward/backward compatibility if (1)
> there is better separation between the object being serialized and the
> object used in application logic (think of the "detached" classes) and (2)
> gryo serializers for 3.0 have greater intelligence around internal
> compatibility handling.
>
> Anyway, I was mostly writing this as notes to myself in preparation for
> 3.3.0, but I'd be happy to hear any thoughts anyone might have on this.
>