You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by Marko Rodriguez <ok...@gmail.com> on 2017/06/26 14:56:26 UTC
[DISCUSS] GraphSON as the future of all serialization.
Hello,
This is a “big idea” in that I don’t see it being helpful for TinkerPop3, but something to consider for TinkerPop4.
We currently support two “internal” serialization mechanisms — Gryo and GraphSON. We also support GraphML but this is primarily for data reading/writing (interoperability with other graph frameworks).
The reason we have Gryo is that its “fast” and good for things that are Java-to-Java (or internal to the Gremlin VM like OLAP serialization).
However:
1. With the development of the Gremlin language variants outside the JavaVM (e.g. Gremlin-C# and Gremlin-Python), GraphSON is the only viable serialization format.
2. With the development of other Gremlin virtual machines (e.g. CosmosDB’s Gremlin .NET implementation), GraphSON is the only viable serialization format.
I think we should work to get rid of Gryo for TinkerPop4 and make GraphSON the default/standard/universal serialization mechanism.
To do this, we need to make it “fast.”
I saw an excellent talk by the ArangoDB guys at GraphDay about their immutable JSON serialization format. In short, they can have a binary JSON representation and can get data from it without having to turn it into a nested-Map structure. In this way, they can do fast lookup operations on the byte stream.
With that, we could learn from their efforts and develop a class along the lines of:
GraphSONByteStream.toJSON() -> yields the String { [ ] } JSON representation.
GraphSONByteStream.toMap() -> yields the nested Map structure we are familiar with in TinkerPop.
GraphSONByteStream.get(“/id”) -> does a random access lookup into the byte stream so we don’t have to deserialize — all byte offset based.
In short, Gryo is too hard to manage and too Java specific. In the future, we should look to making an ultra fast GraphSON serializer/deserializer along with ensuring an elegant, self consistent representation of all the necessary TinkerPop objects.
Thoughts?,
Marko.
http://markorodriguez.com
Re: [DISCUSS] GraphSON as the future of all serialization.
Posted by Stephen Mallette <sp...@gmail.com>.
In retrospect, we really didn't have a good picture of what IO should have
been for TinkerPop 3.x. The use cases made sense at the time but I don't
think we foresaw GLVs terribly well. Now Gryo is a bit of a burden. Getting
tired of hassling with three serialization formats. At least GraphML is
pretty solid and doesn't take much effort to maintain.
On Mon, Jun 26, 2017 at 10:59 AM, Robert Dale <ro...@gmail.com> wrote:
> Is that https://github.com/arangodb/velocypack ?
>
> Robert Dale
>
> On Mon, Jun 26, 2017 at 10:56 AM, Marko Rodriguez <ok...@gmail.com>
> wrote:
>
> > Hello,
> >
> > This is a “big idea” in that I don’t see it being helpful for TinkerPop3,
> > but something to consider for TinkerPop4.
> >
> > We currently support two “internal” serialization mechanisms — Gryo and
> > GraphSON. We also support GraphML but this is primarily for data
> > reading/writing (interoperability with other graph frameworks).
> >
> > The reason we have Gryo is that its “fast” and good for things that are
> > Java-to-Java (or internal to the Gremlin VM like OLAP serialization).
> >
> > However:
> > 1. With the development of the Gremlin language variants outside
> > the JavaVM (e.g. Gremlin-C# and Gremlin-Python), GraphSON is the only
> > viable serialization format.
> > 2. With the development of other Gremlin virtual machines (e.g.
> > CosmosDB’s Gremlin .NET implementation), GraphSON is the only viable
> > serialization format.
> >
> > I think we should work to get rid of Gryo for TinkerPop4 and make
> GraphSON
> > the default/standard/universal serialization mechanism.
> >
> > To do this, we need to make it “fast.”
> >
> > I saw an excellent talk by the ArangoDB guys at GraphDay about their
> > immutable JSON serialization format. In short, they can have a binary
> JSON
> > representation and can get data from it without having to turn it into a
> > nested-Map structure. In this way, they can do fast lookup operations on
> > the byte stream.
> >
> > With that, we could learn from their efforts and develop a class along
> the
> > lines of:
> >
> > GraphSONByteStream.toJSON() -> yields the String { [ ] } JSON
> > representation.
> > GraphSONByteStream.toMap() -> yields the nested Map structure we
> > are familiar with in TinkerPop.
> > GraphSONByteStream.get(“/id”) -> does a random access lookup into
> > the byte stream so we don’t have to deserialize — all byte offset based.
> >
> > In short, Gryo is too hard to manage and too Java specific. In the
> future,
> > we should look to making an ultra fast GraphSON serializer/deserializer
> > along with ensuring an elegant, self consistent representation of all the
> > necessary TinkerPop objects.
> >
> > Thoughts?,
> > Marko.
> >
> > http://markorodriguez.com
> >
> >
> >
> >
>
Re: [DISCUSS] GraphSON as the future of all serialization.
Posted by Marko Rodriguez <ok...@gmail.com>.
Hi,
> Is that https://github.com/arangodb/velocypack ?
Yea. There is a Java version:
https://github.com/arangodb/java-velocypack <https://github.com/arangodb/java-velocypack>
I haven’t studied it, but the general idea of byteoffset lookups in order to nullify the need full object deserialization is the good idea.
Marko.
>
> Robert Dale
>
> On Mon, Jun 26, 2017 at 10:56 AM, Marko Rodriguez <ok...@gmail.com>
> wrote:
>
>> Hello,
>>
>> This is a “big idea” in that I don’t see it being helpful for TinkerPop3,
>> but something to consider for TinkerPop4.
>>
>> We currently support two “internal” serialization mechanisms — Gryo and
>> GraphSON. We also support GraphML but this is primarily for data
>> reading/writing (interoperability with other graph frameworks).
>>
>> The reason we have Gryo is that its “fast” and good for things that are
>> Java-to-Java (or internal to the Gremlin VM like OLAP serialization).
>>
>> However:
>> 1. With the development of the Gremlin language variants outside
>> the JavaVM (e.g. Gremlin-C# and Gremlin-Python), GraphSON is the only
>> viable serialization format.
>> 2. With the development of other Gremlin virtual machines (e.g.
>> CosmosDB’s Gremlin .NET implementation), GraphSON is the only viable
>> serialization format.
>>
>> I think we should work to get rid of Gryo for TinkerPop4 and make GraphSON
>> the default/standard/universal serialization mechanism.
>>
>> To do this, we need to make it “fast.”
>>
>> I saw an excellent talk by the ArangoDB guys at GraphDay about their
>> immutable JSON serialization format. In short, they can have a binary JSON
>> representation and can get data from it without having to turn it into a
>> nested-Map structure. In this way, they can do fast lookup operations on
>> the byte stream.
>>
>> With that, we could learn from their efforts and develop a class along the
>> lines of:
>>
>> GraphSONByteStream.toJSON() -> yields the String { [ ] } JSON
>> representation.
>> GraphSONByteStream.toMap() -> yields the nested Map structure we
>> are familiar with in TinkerPop.
>> GraphSONByteStream.get(“/id”) -> does a random access lookup into
>> the byte stream so we don’t have to deserialize — all byte offset based.
>>
>> In short, Gryo is too hard to manage and too Java specific. In the future,
>> we should look to making an ultra fast GraphSON serializer/deserializer
>> along with ensuring an elegant, self consistent representation of all the
>> necessary TinkerPop objects.
>>
>> Thoughts?,
>> Marko.
>>
>> http://markorodriguez.com
>>
>>
>>
>>
Re: [DISCUSS] GraphSON as the future of all serialization.
Posted by Robert Dale <ro...@gmail.com>.
Is that https://github.com/arangodb/velocypack ?
Robert Dale
On Mon, Jun 26, 2017 at 10:56 AM, Marko Rodriguez <ok...@gmail.com>
wrote:
> Hello,
>
> This is a “big idea” in that I don’t see it being helpful for TinkerPop3,
> but something to consider for TinkerPop4.
>
> We currently support two “internal” serialization mechanisms — Gryo and
> GraphSON. We also support GraphML but this is primarily for data
> reading/writing (interoperability with other graph frameworks).
>
> The reason we have Gryo is that its “fast” and good for things that are
> Java-to-Java (or internal to the Gremlin VM like OLAP serialization).
>
> However:
> 1. With the development of the Gremlin language variants outside
> the JavaVM (e.g. Gremlin-C# and Gremlin-Python), GraphSON is the only
> viable serialization format.
> 2. With the development of other Gremlin virtual machines (e.g.
> CosmosDB’s Gremlin .NET implementation), GraphSON is the only viable
> serialization format.
>
> I think we should work to get rid of Gryo for TinkerPop4 and make GraphSON
> the default/standard/universal serialization mechanism.
>
> To do this, we need to make it “fast.”
>
> I saw an excellent talk by the ArangoDB guys at GraphDay about their
> immutable JSON serialization format. In short, they can have a binary JSON
> representation and can get data from it without having to turn it into a
> nested-Map structure. In this way, they can do fast lookup operations on
> the byte stream.
>
> With that, we could learn from their efforts and develop a class along the
> lines of:
>
> GraphSONByteStream.toJSON() -> yields the String { [ ] } JSON
> representation.
> GraphSONByteStream.toMap() -> yields the nested Map structure we
> are familiar with in TinkerPop.
> GraphSONByteStream.get(“/id”) -> does a random access lookup into
> the byte stream so we don’t have to deserialize — all byte offset based.
>
> In short, Gryo is too hard to manage and too Java specific. In the future,
> we should look to making an ultra fast GraphSON serializer/deserializer
> along with ensuring an elegant, self consistent representation of all the
> necessary TinkerPop objects.
>
> Thoughts?,
> Marko.
>
> http://markorodriguez.com
>
>
>
>