You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by sg...@orange.com on 2015/12/01 13:58:04 UTC
Bug in Gremlin driver serialization
Hi everybody,
Our company is developing a system based on Gremlin server and we are facing an issue for which, I think, I have identified the root cause.
Our problem occurs when requesting a graph that returns beyond a certain amount of data, actually 4096 bytes, through a websocket connection.
I think that the issue is located in the serialization mechanism, in the method serializeResponseAsBinary(...) of the org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0 class.
In this method, we create a local java.io.ByteArrayOutputStream object which is then passed to the constructor of a new instance of an org.apache.tinkerpop.shaded.kryo.io.Output. Subsequent operations serialize data in this Output object and at the end the serializeResponseAsBinary(...) method returns a newly allocated io.netty.buffer.ByteBuf object containing the bytes accumulated in the Output instance. This encoded response message is built as this:
ByteBuf encodedMessage = null;
final OutputStream baos = new ByteArrayOutputStream();
final Output output = new Output(baos);
...
encodedMessage = allocator.buffer((int) output.total());
encodedMessage.writeBytes(output.toBytes());
...
return encodedMessage;
The problem is that the output.toBytes() method only returns the bytes contained in an internal recyclable buffer and does not take into account the bytes that may have been previously flushed into the internal OutputStream (in our case the ByteArrayOutputStream provided to the constructor). Thus, at the end, the encoded response message contains partial data that appear as a corrupted stream to the deserialization process.
What do you think about this problem ? Is there anyone that already faced this issue ? Do you think it would be possible to fix it quickly ?
Regards,
[http://www.orange.com/sirius/logos_mail/orange_logo.gif]<http://www.orange.com/>
Stéphane Gobancé
Prestataire externe
GyGraph / PnS
Tél : 04 97 46 05 65
sgobance.ext@orange.com
Altran pour Orange
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorization.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
Thank you.
Re: Bug in Gremlin driver serialization
Posted by Dylan Millikin <dy...@gmail.com>.
nice catch Jason!
On Tue, Dec 1, 2015 at 3:55 PM, Jason Plurad <pl...@gmail.com> wrote:
> Isn't this fixed already? See Stephen's commits on September 11, 2015.
>
>
> https://github.com/apache/incubator-tinkerpop/commits/3.0.2-incubating/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/ser/GryoMessageSerializerV1d0.java
>
> On Tue, Dec 1, 2015 at 9:50 AM, Dylan Millikin <dy...@gmail.com>
> wrote:
>
> > Hi Stéphane,
> >
> > Thanks for the feedback. I haven't had the time to take a deeper look
> but a
> > 5mn googlefu gave me this which might (or might not) be a lead:
> > https://github.com/EsotericSoftware/kryo/issues/336
> > If we set a failing test we can maybe see if those changes are relevant
> or
> > not.
> >
> > Do you want to give those a try locally and report back? Also if you
> don't
> > mind creating a JIRA issue for this that would be great.
> >
> > Cheers,
> >
> > Dylan.
> >
> >
> >
> >
> >
> > On Tue, Dec 1, 2015 at 1:58 PM, <sg...@orange.com> wrote:
> >
> > > Hi everybody,
> > >
> > >
> > >
> > > Our company is developing a system based on Gremlin server and we are
> > > facing an issue for which, I think, I have identified the root cause.
> > >
> > > Our problem occurs when requesting a graph that returns beyond a
> certain
> > > amount of data, actually 4096 bytes, through a websocket connection.
> > >
> > > I think that the issue is located in the serialization mechanism, in
> the
> > > method *serializeResponseAsBinary*(…) of the
> > *org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0
> > > class*.
> > >
> > >
> > >
> > > In this method, we create a local *java.io.ByteArrayOutputStream*
> object
> > > which is then passed to the constructor of a new instance of an
> > > *org.apache.tinkerpop.shaded.kryo.io.Output*. Subsequent operations
> > > serialize data in this Output object and at the end the
> > > *serializeResponseAsBinary*(…) method returns a newly allocated
> > > *io.netty.buffer.ByteBuf* object containing the bytes accumulated in
> the
> > > Output instance. This encoded response message is built as this:
> > >
> > >
> > >
> > > ByteBuf encodedMessage = null;
> > >
> > > final OutputStream baos = new ByteArrayOutputStream();
> > >
> > > final Output output = new Output(baos);
> > >
> > > …
> > >
> > > encodedMessage = allocator.buffer((*int*) output.total());
> > >
> > > encodedMessage.writeBytes(*output**.toBytes()*);
> > >
> > > …
> > >
> > > return encodedMessage;
> > >
> > >
> > >
> > > The problem is that the *output.toBytes()* method only returns the
> bytes
> > > contained in an internal recyclable buffer and does not take into
> account
> > > the bytes that may have been previously flushed into the internal
> > > OutputStream (in our case the ByteArrayOutputStream provided to the
> > > constructor). Thus, at the end, the encoded response message contains
> > > partial data that appear as a corrupted stream to the deserialization
> > > process.
> > >
> > >
> > >
> > > What do you think about this problem ? Is there anyone that already
> faced
> > > this issue ? Do you think it would be possible to fix it quickly ?
> > >
> > >
> > >
> > > Regards,
> > >
> > > [image: http://www.orange.com/sirius/logos_mail/orange_logo.gif]
> > > <http://www.orange.com/>
> > >
> > > *Stéphane Gobancé *
> > >
> > > *Prestataire externe*
> > >
> > > GyGraph / PnS
> > >
> > > Tél : 04 97 46 05 65
> > >
> > > *sgobance.ext@orange.com <sg...@orange.com>*
> > >
> > > Altran pour Orange
> > >
> > >
> > >
> > >
> >
> _________________________________________________________________________________________________________________________
> > >
> > > Ce message et ses pieces jointes peuvent contenir des informations
> > confidentielles ou privilegiees et ne doivent donc
> > > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
> > recu ce message par erreur, veuillez le signaler
> > > a l'expediteur et le detruire ainsi que les pieces jointes. Les
> messages
> > electroniques etant susceptibles d'alteration,
> > > France Telecom - Orange decline toute responsabilite si ce message a
> ete
> > altere, deforme ou falsifie. Merci
> > >
> > > This message and its attachments may contain confidential or privileged
> > information that may be protected by law;
> > > they should not be distributed, used or copied without authorization.
> > > If you have received this email in error, please notify the sender and
> > delete this message and its attachments.
> > > As emails may be altered, France Telecom - Orange shall not be liable
> if
> > this message was modified, changed or falsified.
> > > Thank you.
> > >
> > >
> >
>
>
>
> --
> Have a good one,
> Jason
>
Re: Bug in Gremlin driver serialization
Posted by Jason Plurad <pl...@gmail.com>.
Isn't this fixed already? See Stephen's commits on September 11, 2015.
https://github.com/apache/incubator-tinkerpop/commits/3.0.2-incubating/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/ser/GryoMessageSerializerV1d0.java
On Tue, Dec 1, 2015 at 9:50 AM, Dylan Millikin <dy...@gmail.com>
wrote:
> Hi Stéphane,
>
> Thanks for the feedback. I haven't had the time to take a deeper look but a
> 5mn googlefu gave me this which might (or might not) be a lead:
> https://github.com/EsotericSoftware/kryo/issues/336
> If we set a failing test we can maybe see if those changes are relevant or
> not.
>
> Do you want to give those a try locally and report back? Also if you don't
> mind creating a JIRA issue for this that would be great.
>
> Cheers,
>
> Dylan.
>
>
>
>
>
> On Tue, Dec 1, 2015 at 1:58 PM, <sg...@orange.com> wrote:
>
> > Hi everybody,
> >
> >
> >
> > Our company is developing a system based on Gremlin server and we are
> > facing an issue for which, I think, I have identified the root cause.
> >
> > Our problem occurs when requesting a graph that returns beyond a certain
> > amount of data, actually 4096 bytes, through a websocket connection.
> >
> > I think that the issue is located in the serialization mechanism, in the
> > method *serializeResponseAsBinary*(…) of the
> *org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0
> > class*.
> >
> >
> >
> > In this method, we create a local *java.io.ByteArrayOutputStream* object
> > which is then passed to the constructor of a new instance of an
> > *org.apache.tinkerpop.shaded.kryo.io.Output*. Subsequent operations
> > serialize data in this Output object and at the end the
> > *serializeResponseAsBinary*(…) method returns a newly allocated
> > *io.netty.buffer.ByteBuf* object containing the bytes accumulated in the
> > Output instance. This encoded response message is built as this:
> >
> >
> >
> > ByteBuf encodedMessage = null;
> >
> > final OutputStream baos = new ByteArrayOutputStream();
> >
> > final Output output = new Output(baos);
> >
> > …
> >
> > encodedMessage = allocator.buffer((*int*) output.total());
> >
> > encodedMessage.writeBytes(*output**.toBytes()*);
> >
> > …
> >
> > return encodedMessage;
> >
> >
> >
> > The problem is that the *output.toBytes()* method only returns the bytes
> > contained in an internal recyclable buffer and does not take into account
> > the bytes that may have been previously flushed into the internal
> > OutputStream (in our case the ByteArrayOutputStream provided to the
> > constructor). Thus, at the end, the encoded response message contains
> > partial data that appear as a corrupted stream to the deserialization
> > process.
> >
> >
> >
> > What do you think about this problem ? Is there anyone that already faced
> > this issue ? Do you think it would be possible to fix it quickly ?
> >
> >
> >
> > Regards,
> >
> > [image: http://www.orange.com/sirius/logos_mail/orange_logo.gif]
> > <http://www.orange.com/>
> >
> > *Stéphane Gobancé *
> >
> > *Prestataire externe*
> >
> > GyGraph / PnS
> >
> > Tél : 04 97 46 05 65
> >
> > *sgobance.ext@orange.com <sg...@orange.com>*
> >
> > Altran pour Orange
> >
> >
> >
> >
> _________________________________________________________________________________________________________________________
> >
> > Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
> > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
> recu ce message par erreur, veuillez le signaler
> > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
> > France Telecom - Orange decline toute responsabilite si ce message a ete
> altere, deforme ou falsifie. Merci
> >
> > This message and its attachments may contain confidential or privileged
> information that may be protected by law;
> > they should not be distributed, used or copied without authorization.
> > If you have received this email in error, please notify the sender and
> delete this message and its attachments.
> > As emails may be altered, France Telecom - Orange shall not be liable if
> this message was modified, changed or falsified.
> > Thank you.
> >
> >
>
--
Have a good one,
Jason
Re: Bug in Gremlin driver serialization
Posted by Dylan Millikin <dy...@gmail.com>.
Hi Stéphane,
Thanks for the feedback. I haven't had the time to take a deeper look but a
5mn googlefu gave me this which might (or might not) be a lead:
https://github.com/EsotericSoftware/kryo/issues/336
If we set a failing test we can maybe see if those changes are relevant or
not.
Do you want to give those a try locally and report back? Also if you don't
mind creating a JIRA issue for this that would be great.
Cheers,
Dylan.
On Tue, Dec 1, 2015 at 1:58 PM, <sg...@orange.com> wrote:
> Hi everybody,
>
>
>
> Our company is developing a system based on Gremlin server and we are
> facing an issue for which, I think, I have identified the root cause.
>
> Our problem occurs when requesting a graph that returns beyond a certain
> amount of data, actually 4096 bytes, through a websocket connection.
>
> I think that the issue is located in the serialization mechanism, in the
> method *serializeResponseAsBinary*(…) of the *org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0
> class*.
>
>
>
> In this method, we create a local *java.io.ByteArrayOutputStream* object
> which is then passed to the constructor of a new instance of an
> *org.apache.tinkerpop.shaded.kryo.io.Output*. Subsequent operations
> serialize data in this Output object and at the end the
> *serializeResponseAsBinary*(…) method returns a newly allocated
> *io.netty.buffer.ByteBuf* object containing the bytes accumulated in the
> Output instance. This encoded response message is built as this:
>
>
>
> ByteBuf encodedMessage = null;
>
> final OutputStream baos = new ByteArrayOutputStream();
>
> final Output output = new Output(baos);
>
> …
>
> encodedMessage = allocator.buffer((*int*) output.total());
>
> encodedMessage.writeBytes(*output**.toBytes()*);
>
> …
>
> return encodedMessage;
>
>
>
> The problem is that the *output.toBytes()* method only returns the bytes
> contained in an internal recyclable buffer and does not take into account
> the bytes that may have been previously flushed into the internal
> OutputStream (in our case the ByteArrayOutputStream provided to the
> constructor). Thus, at the end, the encoded response message contains
> partial data that appear as a corrupted stream to the deserialization
> process.
>
>
>
> What do you think about this problem ? Is there anyone that already faced
> this issue ? Do you think it would be possible to fix it quickly ?
>
>
>
> Regards,
>
> [image: http://www.orange.com/sirius/logos_mail/orange_logo.gif]
> <http://www.orange.com/>
>
> *Stéphane Gobancé *
>
> *Prestataire externe*
>
> GyGraph / PnS
>
> Tél : 04 97 46 05 65
>
> *sgobance.ext@orange.com <sg...@orange.com>*
>
> Altran pour Orange
>
>
>
> _________________________________________________________________________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
> France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci
>
> This message and its attachments may contain confidential or privileged information that may be protected by law;
> they should not be distributed, used or copied without authorization.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
> Thank you.
>
>