You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@thrift.apache.org by Nobuaki Sukegawa <ns...@gmail.com> on 2015/12/01 16:04:27 UTC
Re: Serializing struct using c_glib into memory buffer Help!
Hi GZ,
> string laid out as if it was going out "over-the-wire"
That's the whole point of memory buffer and it is supposed to work for your
use case.
Even with the correct length you don't see the string ?
You may want to just try sending the data of length reported by *read*
method and deserializing on the other side of the wire.
On Tue, Dec 1, 2015 at 4:00 AM GZ <gz...@zygmund.net> wrote:
> Hi Nobuaki,
>
> Thanks... I will keep an eye out for the fix. As you pointed out, using
> read returns the correct length (as far as I can tell) and I am able to
> deserialise all the data into a new object correctly. Thanks! However, when
> I look at the memory buffer, I still do not see the string. I suspect that
> the buffer contains a pointer to the string, not the string laid out as if
> it was going out "over-the-wire".
>
> Mayber I am accessing the buffer incorrectly or using the wrong transport?
> Ultimately I want to be able to take the whole buffer and send it to Kafka
> as a message for desrialisation by other components.
>
> Your help is much appreciated!
>
> George
>
>
>
>
> Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > Hi GZ,
> >
> > You're absolutely right about both "&" and the length/contents.
> > It turned out that it is a bug: as you pointed out the write method
> ignores
> > some bytes written.
> > So the data is written and ready to be deserialized but the reported
> length
> > is wrong.
> > If you happen to be building Thrift from the source, the fix will be
> > available on master branch soon.
> >
> > A very dirty and non-performant workaround is to deserialize the buffer
> > only to know the byte length before sending:
> > wrong_len = cls->write(THRIFT_STRUCT(pData2), ...);
> > correct_len = cls->read(THRIFT_STRUCT(pData2), ...);
> > you'll see correct byte length since read is not affected by this bug.
> >
> > On Sun, Nov 29, 2015 at 4:22 AM GZ <gz...@zygmund.net> wrote:
> >
> > > Hi Nobuaki,
> > >
> > > I tried this originally but write is expecting a ThriftStruct* rather
> than
> > > a ThriftStruct. I have tried with both &pData->parent and pData.. the
> > > results are pretty much the same. Below is a Hex printout of the memory
> > > buffer (19 bytes).
> > >
> > > &pData->parent
> > > 01FB5040 | 1081FB01 00000000 52000000 80000000 | ........R.......
> > > 01FB5050 | 010000
> > >
> > > pData
> > > 01EA4040 | 1071EA01 00000000 52000000 80000000 | .q......R.......
> > > 01EA4050 | 010000
> > >
> > >
> > > I would expect to see the "XXXXXXX" string serialised into this buffer
> but
> > > it is not.
> > >
> > > George
> > >
> > >
> > >
> > > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > > Hi GZ,
> > > >
> > > > Can you try with "pData->parent" part applied too ?
> > > > Sorry I should have mentioned this too.
> > > >
> > > >
> > > > On Sat, Nov 28, 2015 at 8:25 AM GZ <gz...@zygmund.net> wrote:
> > > >
> > > > > Hi Nobuaki,
> > > > >
> > > > > Yes - the rdata_write was in the generated c file not the header.
> > > > >
> > > > > I made the changes you suggested, it was just what I was looking
> for. I
> > > > > have repeated the corrected code to help others but unfortunately
> the
> > > > > original problem persists..
> > > > >
> > > > >
> > > > > tbuffer = g_object_new (THRIFT_TYPE_MEMORY_BUFFER, "buf_size",
> > > 1000,
> > > > > NULL);
> > > > > if(tbuffer){
> > > > > transport = THRIFT_TRANSPORT(tbuffer);
> > > > > thrift_transport_open (transport, &error);
> > > > >
> > > > > protocol = THRIFT_PROTOCOL(g_object_new
> > > > > (THRIFT_TYPE_BINARY_PROTOCOL, "transport", transport, NULL));
> > > > > if(protocol){
> > > > > rdata* pData;
> > > > > pData = g_object_new (TYPE_RDATA,"flag",1,
> > > > > "id",27,
> > > > > "start",12000,
> > > > > "done",345435,
> > > > > "notif",34455,
> > > > >
> > > > > "spid","XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",NULL);
> > > > > if(pData){
> > > > > ThriftStructClass* cls =
> > > > > THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData));
> > > > > len = cls->write(pData,protocol,&error);
> > > > >
> > > > > printf ("%d Bytes written to buffer\n", len);
> > > > >
> > > > > if(tbuffer->buf != NULL){
> > > > > printHex((const void *)tbuffer->buf, 100);
> > > > > }
> > > > > g_object_unref (pData);
> > > > > }
> > > > > g_object_unref (protocol);
> > > > > }
> > > > > g_object_unref(tbuffer);
> > > > > }
> > > > >
> > > > >
> > > > > The returned buffer length still appears incorrect and the string
> does
> > > not
> > > > > appear to be serialised into the buffer.
> > > > >
> > > > > I can track the writes through the writing function and I notice
> that
> > > for
> > > > > the spid string value
> > > > >
> > > > > thrift_protocol_write_field_begin returns 3 bytes
> > > > > thrift_protocol_write_string returns 50 bytes
> > > > > thrift_protocol_write_field_end returns 0
> > > > >
> > > > > the 50 bytes never get added to the length because it is overriden
> by
> > > the
> > > > > 0. This is a snippit of the code..
> > > > >
> > > > > if ((ret = thrift_protocol_write_field_begin (protocol, "spid",
> > > > > T_STRING, 6, error)) < 0)
> > > > > return -1;
> > > > > xfer += ret;
> > > > > if ((ret = thrift_protocol_write_string (protocol,
> this_object->spid,
> > > > > error)) < 0)
> > > > > return -1;
> > > > > if ((ret = thrift_protocol_write_field_end (protocol, error)) <
> 0)
> > > > > return -1;
> > > > > xfer += ret;
> > > > >
> > > > > The result is that the write function returns 19 bytes and when I
> view
> > > the
> > > > > buffer I see no string. I guess I must still be missing
> something...
> > > The
> > > > > spid string is 46 characters long. I would expect to see this
> > > reflected in
> > > > > the length of the buffer.
> > > > >
> > > > > The idl record is
> > > > >
> > > > > enum Status {
> > > > > ONE = 1,
> > > > > TWO = 2,
> > > > > THREE = 4,
> > > > > }
> > > > >
> > > > > struct rdata {
> > > > > 1: Status flag,
> > > > > 2: i16 id,
> > > > > 3: i64 start,
> > > > > 4: i64 done,
> > > > > 5: i64 notif,
> > > > > 6: string spid
> > > > > }
> > > > >
> > > > > Any idea what might be happenning?
> > > > >
> > > > > Thanks again...
> > > > >
> > > > > George
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > > > > It is strange, the generated header has rdata_write ?
> > > > > > Seeing generated code, it should be something like this:
> > > > > >
> > > > > > ThriftStructClass* cls =
> THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData));
> > > > > > cls->write(pData->parent, protocol, &error);
> > > > > >
> > > > > > On Thu, Nov 26, 2015 at 8:03 AM GZ <gz...@zygmund.net> wrote:
> > > > > >
> > > > > > > I am new to thrift and have been trying to serialize a thrift
> > > structure
> > > > > > > using c but have been unable to get it to work. I want to
> serialize
> > > > > using
> > > > > > > the binary protocol into a memory buffer. I do not need RPC, I
> > > will be
> > > > > > > passing the message into Kafka. Does anyone know the correct
> way
> > > to do
> > > > > this?
> > > > > > >
> > > > > > > I have looked through all the examples and test code and have
> come
> > > up
> > > > > with
> > > > > > > the following. It compiles, but is not working correctly. I am
> > > clearly
> > > > > not
> > > > > > > doing it correctly. rdata_write is a method generated by
> thrift.
> > > > > > >
> > > > > > > GError *error = NULL;
> > > > > > > ThriftTransport *transport = NULL;
> > > > > > > ThriftProtocol *protocol = NULL;
> > > > > > >
> > > > > > > transport=g_object_new (THRIFT_TYPE_MEMORY_BUFFER,
> "buf_size",
> > > 100,
> > > > > > > NULL);
> > > > > > > if(transport){
> > > > > > > thrift_transport_open (transport, &error);
> > > > > > > protocol = g_object_new (THRIFT_TYPE_BINARY_PROTOCOL,
> > > > > "transport",
> > > > > > > transport,NULL);
> > > > > > > if(protocol){
> > > > > > > rdata* pData = g_object_new (TYPE_RDATA,"flag",1,
> > > > > > > "id",27,
> > > > > > > "start",12000,
> > > > > > > "done",345435,
> > > > > > > "notif",34455,
> > > > > > > "spid","This is the
> > > SPID",NULL);
> > > > > > > if(pData){
> > > > > > >
> > > > > > > /* wanted something like
> > > obj->write(protocol,&error)
> > > > > > > But how to get object? pData does not have
> write
> > > > > method
> > > > > > > */
> > > > > > >
> > > > > > > gint32 len =
> > > > > > > rdata_write(pData,(ThriftProtocol*)protocol,&error);
> > > > > > >
> > > > > > > // Print Content of Buffer
> > > > > > > ThriftMemoryBuffer* pMem =
> > > > > (ThriftMemoryBuffer*)transport;
> > > > > > > if(pMem->buf != NULL){
> > > > > > > printHex((const void *)pMem->buf, len);
> > > > > > > }
> > > > > > > g_object_unref (pData);
> > > > > > > }
> > > > > > > g_object_unref (protocol);
> > > > > > > }
> > > > > > > g_object_unref(transport);
> > > > > > > }
> > > > > > >
> > > > > > > Error handling and support functions skipped for brevity.
> > > > > > >
> > > > > > > Any help appreciated
> > > > > > >
> > > > >
> > >
>
Re: Serializing struct using c_glib into memory buffer Help!
Posted by GZ <gz...@zygmund.net>.
Hi Nobuaki,
I had not realised I was dealing with a GByteArray*. When I use tbuffer->buf->data I can now see the string in the buffer but there are problems. The buffer has a null at the start and is missing all the other parts of the data other than the string. Here is the hex output... (compare to the output in my previous message )
00615110 | 00585858 58585858 58585858 58585858 | .XXXXXXXXXXXXXXX
00615120 | 58585858 58585858 58585858 58005858 | XXXXXXXXXXXXX.XX
00615130 | 58005800 00585858 58585858 00580000 | X.X..XXXXXXX.X..
00615140 | 58585858 58585800 58000058 58585858 | XXXXXXX.X..XXXXX
00615150 | 58580058 00005800 58000058 58580058 | XX.X..X.X..XXX.X
00615160 | 0000
I am not familiar with GByteArrays but I did some tests with g_bytes_get_data ( which returned the same pointer as tbuffer->buf->data ) and g_bytes_get_size ( which returned an incorrect large value). I think there is something wrong...
What do you think?
I noticed you have committted an update to the bug you identified - I will update my code to see if that helps with this...
Thanks for your support...
George
Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> Hi GZ,
>
> (const void *)tbuffer->buf seems problematic: it's a GByteArray*.
> tbuffer->buf->data should point to the data.
>
> On Sun, Dec 6, 2015 at 7:10 AM GZ <gz...@zygmund.net> wrote:
>
> > Hi Nobuaki,
> >
> > That was my understanding too. So when I write the object I would expect
> > to see the data serialised into the memory buffer... in tbuffer->buf But I
> > dont. It would be hard to miss the long "XXXXX.." string.
> >
> > The data must be correctly stored because I was able load the data into a
> > new object correctly.
> >
> > I did try your suggestion and try deserialise the buffer after it had been
> > moved but this did not work, the object was empty.
> >
> > Its very frustrating. I tried the same process using the cpp library and
> > it works!
> >
> > 013394F0 | 08000100 00000206 0002012C 0A000300 | ...........,....
> > 01339500 | 00000000 63BF280A 00040000 00000063 | ....c.(........c
> > 01339510 | C0F20A00 05000000 000063BF BE0B0006 | ..........c.....
> > 01339520 | 0000001D 58585858 58585858 58585858 | ....XXXXXXXXXXXX
> > 01339530 | 58585858 58585858 58585858 58585858 | XXXXXXXXXXXXXXXX
> > 01339540 | 5800 | X.
> >
> > I think I must be getting the buffer incorrectly because the output above
> > is completely different for the buffer I get in the c_glib version
> > (identical object).
> >
> > Any Ideas ?!
> >
> > Thanks again
> > George
> >
> >
> >
> > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > Hi GZ,
> > >
> > > > string laid out as if it was going out "over-the-wire"
> > > That's the whole point of memory buffer and it is supposed to work for
> > your
> > > use case.
> > > Even with the correct length you don't see the string ?
> > > You may want to just try sending the data of length reported by *read*
> > > method and deserializing on the other side of the wire.
> > >
> > > On Tue, Dec 1, 2015 at 4:00 AM GZ <gz...@zygmund.net> wrote:
> > >
> > > > Hi Nobuaki,
> > > >
> > > > Thanks... I will keep an eye out for the fix. As you pointed out, using
> > > > read returns the correct length (as far as I can tell) and I am able to
> > > > deserialise all the data into a new object correctly. Thanks! However,
> > when
> > > > I look at the memory buffer, I still do not see the string. I suspect
> > that
> > > > the buffer contains a pointer to the string, not the string laid out
> > as if
> > > > it was going out "over-the-wire".
> > > >
> > > > Mayber I am accessing the buffer incorrectly or using the wrong
> > transport?
> > > > Ultimately I want to be able to take the whole buffer and send it to
> > Kafka
> > > > as a message for desrialisation by other components.
> > > >
> > > > Your help is much appreciated!
> > > >
> > > > George
> > > >
> > > >
> > > >
> > > >
> > > > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > > > Hi GZ,
> > > > >
> > > > > You're absolutely right about both "&" and the length/contents.
> > > > > It turned out that it is a bug: as you pointed out the write method
> > > > ignores
> > > > > some bytes written.
> > > > > So the data is written and ready to be deserialized but the reported
> > > > length
> > > > > is wrong.
> > > > > If you happen to be building Thrift from the source, the fix will be
> > > > > available on master branch soon.
> > > > >
> > > > > A very dirty and non-performant workaround is to deserialize the
> > buffer
> > > > > only to know the byte length before sending:
> > > > > wrong_len = cls->write(THRIFT_STRUCT(pData2), ...);
> > > > > correct_len = cls->read(THRIFT_STRUCT(pData2), ...);
> > > > > you'll see correct byte length since read is not affected by this
> > bug.
> > > > >
> > > > > On Sun, Nov 29, 2015 at 4:22 AM GZ <gz...@zygmund.net> wrote:
> > > > >
> > > > > > Hi Nobuaki,
> > > > > >
> > > > > > I tried this originally but write is expecting a ThriftStruct*
> > rather
> > > > than
> > > > > > a ThriftStruct. I have tried with both &pData->parent and pData..
> > the
> > > > > > results are pretty much the same. Below is a Hex printout of the
> > memory
> > > > > > buffer (19 bytes).
> > > > > >
> > > > > > &pData->parent
> > > > > > 01FB5040 | 1081FB01 00000000 52000000 80000000 | ........R.......
> > > > > > 01FB5050 | 010000
> > > > > >
> > > > > > pData
> > > > > > 01EA4040 | 1071EA01 00000000 52000000 80000000 | .q......R.......
> > > > > > 01EA4050 | 010000
> > > > > >
> > > > > >
> > > > > > I would expect to see the "XXXXXXX" string serialised into this
> > buffer
> > > > but
> > > > > > it is not.
> > > > > >
> > > > > > George
> > > > > >
> > > > > >
> > > > > >
> > > > > > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > > > > > Hi GZ,
> > > > > > >
> > > > > > > Can you try with "pData->parent" part applied too ?
> > > > > > > Sorry I should have mentioned this too.
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Nov 28, 2015 at 8:25 AM GZ <gz...@zygmund.net> wrote:
> > > > > > >
> > > > > > > > Hi Nobuaki,
> > > > > > > >
> > > > > > > > Yes - the rdata_write was in the generated c file not the
> > header.
> > > > > > > >
> > > > > > > > I made the changes you suggested, it was just what I was
> > looking
> > > > for. I
> > > > > > > > have repeated the corrected code to help others but
> > unfortunately
> > > > the
> > > > > > > > original problem persists..
> > > > > > > >
> > > > > > > >
> > > > > > > > tbuffer = g_object_new (THRIFT_TYPE_MEMORY_BUFFER,
> > "buf_size",
> > > > > > 1000,
> > > > > > > > NULL);
> > > > > > > > if(tbuffer){
> > > > > > > > transport = THRIFT_TRANSPORT(tbuffer);
> > > > > > > > thrift_transport_open (transport, &error);
> > > > > > > >
> > > > > > > > protocol = THRIFT_PROTOCOL(g_object_new
> > > > > > > > (THRIFT_TYPE_BINARY_PROTOCOL, "transport", transport, NULL));
> > > > > > > > if(protocol){
> > > > > > > > rdata* pData;
> > > > > > > > pData = g_object_new (TYPE_RDATA,"flag",1,
> > > > > > > > "id",27,
> > > > > > > > "start",12000,
> > > > > > > > "done",345435,
> > > > > > > > "notif",34455,
> > > > > > > >
> > > > > > > > "spid","XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",NULL);
> > > > > > > > if(pData){
> > > > > > > > ThriftStructClass* cls =
> > > > > > > > THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData));
> > > > > > > > len = cls->write(pData,protocol,&error);
> > > > > > > >
> > > > > > > > printf ("%d Bytes written to buffer\n", len);
> > > > > > > >
> > > > > > > > if(tbuffer->buf != NULL){
> > > > > > > > printHex((const void *)tbuffer->buf, 100);
> > > > > > > > }
> > > > > > > > g_object_unref (pData);
> > > > > > > > }
> > > > > > > > g_object_unref (protocol);
> > > > > > > > }
> > > > > > > > g_object_unref(tbuffer);
> > > > > > > > }
> > > > > > > >
> > > > > > > >
> > > > > > > > The returned buffer length still appears incorrect and the
> > string
> > > > does
> > > > > > not
> > > > > > > > appear to be serialised into the buffer.
> > > > > > > >
> > > > > > > > I can track the writes through the writing function and I
> > notice
> > > > that
> > > > > > for
> > > > > > > > the spid string value
> > > > > > > >
> > > > > > > > thrift_protocol_write_field_begin returns 3 bytes
> > > > > > > > thrift_protocol_write_string returns 50 bytes
> > > > > > > > thrift_protocol_write_field_end returns 0
> > > > > > > >
> > > > > > > > the 50 bytes never get added to the length because it is
> > overriden
> > > > by
> > > > > > the
> > > > > > > > 0. This is a snippit of the code..
> > > > > > > >
> > > > > > > > if ((ret = thrift_protocol_write_field_begin (protocol,
> > "spid",
> > > > > > > > T_STRING, 6, error)) < 0)
> > > > > > > > return -1;
> > > > > > > > xfer += ret;
> > > > > > > > if ((ret = thrift_protocol_write_string (protocol,
> > > > this_object->spid,
> > > > > > > > error)) < 0)
> > > > > > > > return -1;
> > > > > > > > if ((ret = thrift_protocol_write_field_end (protocol,
> > error)) <
> > > > 0)
> > > > > > > > return -1;
> > > > > > > > xfer += ret;
> > > > > > > >
> > > > > > > > The result is that the write function returns 19 bytes and
> > when I
> > > > view
> > > > > > the
> > > > > > > > buffer I see no string. I guess I must still be missing
> > > > something...
> > > > > > The
> > > > > > > > spid string is 46 characters long. I would expect to see this
> > > > > > reflected in
> > > > > > > > the length of the buffer.
> > > > > > > >
> > > > > > > > The idl record is
> > > > > > > >
> > > > > > > > enum Status {
> > > > > > > > ONE = 1,
> > > > > > > > TWO = 2,
> > > > > > > > THREE = 4,
> > > > > > > > }
> > > > > > > >
> > > > > > > > struct rdata {
> > > > > > > > 1: Status flag,
> > > > > > > > 2: i16 id,
> > > > > > > > 3: i64 start,
> > > > > > > > 4: i64 done,
> > > > > > > > 5: i64 notif,
> > > > > > > > 6: string spid
> > > > > > > > }
> > > > > > > >
> > > > > > > > Any idea what might be happenning?
> > > > > > > >
> > > > > > > > Thanks again...
> > > > > > > >
> > > > > > > > George
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > > > > > > > It is strange, the generated header has rdata_write ?
> > > > > > > > > Seeing generated code, it should be something like this:
> > > > > > > > >
> > > > > > > > > ThriftStructClass* cls =
> > > > THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData));
> > > > > > > > > cls->write(pData->parent, protocol, &error);
> > > > > > > > >
> > > > > > > > > On Thu, Nov 26, 2015 at 8:03 AM GZ <gz...@zygmund.net> wrote:
> > > > > > > > >
> > > > > > > > > > I am new to thrift and have been trying to serialize a
> > thrift
> > > > > > structure
> > > > > > > > > > using c but have been unable to get it to work. I want to
> > > > serialize
> > > > > > > > using
> > > > > > > > > > the binary protocol into a memory buffer. I do not need
> > RPC, I
> > > > > > will be
> > > > > > > > > > passing the message into Kafka. Does anyone know the
> > correct
> > > > way
> > > > > > to do
> > > > > > > > this?
> > > > > > > > > >
> > > > > > > > > > I have looked through all the examples and test code and
> > have
> > > > come
> > > > > > up
> > > > > > > > with
> > > > > > > > > > the following. It compiles, but is not working correctly.
> > I am
> > > > > > clearly
> > > > > > > > not
> > > > > > > > > > doing it correctly. rdata_write is a method generated by
> > > > thrift.
> > > > > > > > > >
> > > > > > > > > > GError *error = NULL;
> > > > > > > > > > ThriftTransport *transport = NULL;
> > > > > > > > > > ThriftProtocol *protocol = NULL;
> > > > > > > > > >
> > > > > > > > > > transport=g_object_new (THRIFT_TYPE_MEMORY_BUFFER,
> > > > "buf_size",
> > > > > > 100,
> > > > > > > > > > NULL);
> > > > > > > > > > if(transport){
> > > > > > > > > > thrift_transport_open (transport, &error);
> > > > > > > > > > protocol = g_object_new
> > (THRIFT_TYPE_BINARY_PROTOCOL,
> > > > > > > > "transport",
> > > > > > > > > > transport,NULL);
> > > > > > > > > > if(protocol){
> > > > > > > > > > rdata* pData = g_object_new
> > (TYPE_RDATA,"flag",1,
> > > > > > > > > > "id",27,
> > > > > > > > > > "start",12000,
> > > > > > > > > > "done",345435,
> > > > > > > > > > "notif",34455,
> > > > > > > > > > "spid","This is the
> > > > > > SPID",NULL);
> > > > > > > > > > if(pData){
> > > > > > > > > >
> > > > > > > > > > /* wanted something like
> > > > > > obj->write(protocol,&error)
> > > > > > > > > > But how to get object? pData does not
> > have
> > > > write
> > > > > > > > method
> > > > > > > > > > */
> > > > > > > > > >
> > > > > > > > > > gint32 len =
> > > > > > > > > > rdata_write(pData,(ThriftProtocol*)protocol,&error);
> > > > > > > > > >
> > > > > > > > > > // Print Content of Buffer
> > > > > > > > > > ThriftMemoryBuffer* pMem =
> > > > > > > > (ThriftMemoryBuffer*)transport;
> > > > > > > > > > if(pMem->buf != NULL){
> > > > > > > > > > printHex((const void *)pMem->buf, len);
> > > > > > > > > > }
> > > > > > > > > > g_object_unref (pData);
> > > > > > > > > > }
> > > > > > > > > > g_object_unref (protocol);
> > > > > > > > > > }
> > > > > > > > > > g_object_unref(transport);
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > Error handling and support functions skipped for brevity.
> > > > > > > > > >
> > > > > > > > > > Any help appreciated
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
Re: Serializing struct using c_glib into memory buffer Help!
Posted by GZ <gz...@zygmund.net>.
Hi Nobuaki,
Since my last message, I have recompiled thrift with the latest fixes and identified an issue in my code where I was reading the data back in to a new object. Something I was doing there ( maybe releasing values ) messed up the buffer. If I capture the buffer before this it works fine. Thank-you for your help. For completeness I include my updated code.
GError *error = NULL;
gint32 len = 0;
ThriftTransport *transport = NULL;
ThriftProtocol *protocol = NULL;
ThriftMemoryBuffer *tbuffer = NULL;
char* pBuffer = NULL;
tbuffer = g_object_new (THRIFT_TYPE_MEMORY_BUFFER, "buf_size", 1000, NULL);
if(tbuffer){
transport = THRIFT_TRANSPORT(tbuffer);
thrift_transport_open (transport, &error);
if(error){
printf ("ERROR: Open Transport - %s\n", error->message);
g_error_free (error);
exit(1);
}
protocol = THRIFT_PROTOCOL(g_object_new (THRIFT_TYPE_BINARY_PROTOCOL, "transport", transport, NULL));
if(protocol){
rdata* pData;
pData = g_object_new (TYPE_RDATA,"flag",STATUS_INCACHE,
"id",300,
"start",6537000,
"done",6537458,
"notif",6537150,
"spid","XXXXXXXXXXXXXXXXXXXXXXXXXXXXX",NULL);
if(pData){
ThriftStructClass* cls = THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData));
len = cls->write(THRIFT_STRUCT(pData),protocol,&error);
if(error){
printf ("ERROR: %s\n", error->message);
g_error_free (error);
}
/* Print out contents of Memory Buffer */
pBuffer = tbuffer->buf->data;
if(pBuffer != NULL){
printHex(pBuffer,len);
}
g_object_unref (pData);
}
g_object_unref (protocol);
}
g_object_unref(tbuffer);
}
return (EXIT_SUCCESS);
Thanks Again!
George
Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> Hi GZ,
>
> (const void *)tbuffer->buf seems problematic: it's a GByteArray*.
> tbuffer->buf->data should point to the data.
>
> On Sun, Dec 6, 2015 at 7:10 AM GZ <gz...@zygmund.net> wrote:
>
> > Hi Nobuaki,
> >
> > That was my understanding too. So when I write the object I would expect
> > to see the data serialised into the memory buffer... in tbuffer->buf But I
> > dont. It would be hard to miss the long "XXXXX.." string.
> >
> > The data must be correctly stored because I was able load the data into a
> > new object correctly.
> >
> > I did try your suggestion and try deserialise the buffer after it had been
> > moved but this did not work, the object was empty.
> >
> > Its very frustrating. I tried the same process using the cpp library and
> > it works!
> >
> > 013394F0 | 08000100 00000206 0002012C 0A000300 | ...........,....
> > 01339500 | 00000000 63BF280A 00040000 00000063 | ....c.(........c
> > 01339510 | C0F20A00 05000000 000063BF BE0B0006 | ..........c.....
> > 01339520 | 0000001D 58585858 58585858 58585858 | ....XXXXXXXXXXXX
> > 01339530 | 58585858 58585858 58585858 58585858 | XXXXXXXXXXXXXXXX
> > 01339540 | 5800 | X.
> >
> > I think I must be getting the buffer incorrectly because the output above
> > is completely different for the buffer I get in the c_glib version
> > (identical object).
> >
> > Any Ideas ?!
> >
> > Thanks again
> > George
> >
> >
> >
> > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > Hi GZ,
> > >
> > > > string laid out as if it was going out "over-the-wire"
> > > That's the whole point of memory buffer and it is supposed to work for
> > your
> > > use case.
> > > Even with the correct length you don't see the string ?
> > > You may want to just try sending the data of length reported by *read*
> > > method and deserializing on the other side of the wire.
> > >
> > > On Tue, Dec 1, 2015 at 4:00 AM GZ <gz...@zygmund.net> wrote:
> > >
> > > > Hi Nobuaki,
> > > >
> > > > Thanks... I will keep an eye out for the fix. As you pointed out, using
> > > > read returns the correct length (as far as I can tell) and I am able to
> > > > deserialise all the data into a new object correctly. Thanks! However,
> > when
> > > > I look at the memory buffer, I still do not see the string. I suspect
> > that
> > > > the buffer contains a pointer to the string, not the string laid out
> > as if
> > > > it was going out "over-the-wire".
> > > >
> > > > Mayber I am accessing the buffer incorrectly or using the wrong
> > transport?
> > > > Ultimately I want to be able to take the whole buffer and send it to
> > Kafka
> > > > as a message for desrialisation by other components.
> > > >
> > > > Your help is much appreciated!
> > > >
> > > > George
> > > >
> > > >
> > > >
> > > >
> > > > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > > > Hi GZ,
> > > > >
> > > > > You're absolutely right about both "&" and the length/contents.
> > > > > It turned out that it is a bug: as you pointed out the write method
> > > > ignores
> > > > > some bytes written.
> > > > > So the data is written and ready to be deserialized but the reported
> > > > length
> > > > > is wrong.
> > > > > If you happen to be building Thrift from the source, the fix will be
> > > > > available on master branch soon.
> > > > >
> > > > > A very dirty and non-performant workaround is to deserialize the
> > buffer
> > > > > only to know the byte length before sending:
> > > > > wrong_len = cls->write(THRIFT_STRUCT(pData2), ...);
> > > > > correct_len = cls->read(THRIFT_STRUCT(pData2), ...);
> > > > > you'll see correct byte length since read is not affected by this
> > bug.
> > > > >
> > > > > On Sun, Nov 29, 2015 at 4:22 AM GZ <gz...@zygmund.net> wrote:
> > > > >
> > > > > > Hi Nobuaki,
> > > > > >
> > > > > > I tried this originally but write is expecting a ThriftStruct*
> > rather
> > > > than
> > > > > > a ThriftStruct. I have tried with both &pData->parent and pData..
> > the
> > > > > > results are pretty much the same. Below is a Hex printout of the
> > memory
> > > > > > buffer (19 bytes).
> > > > > >
> > > > > > &pData->parent
> > > > > > 01FB5040 | 1081FB01 00000000 52000000 80000000 | ........R.......
> > > > > > 01FB5050 | 010000
> > > > > >
> > > > > > pData
> > > > > > 01EA4040 | 1071EA01 00000000 52000000 80000000 | .q......R.......
> > > > > > 01EA4050 | 010000
> > > > > >
> > > > > >
> > > > > > I would expect to see the "XXXXXXX" string serialised into this
> > buffer
> > > > but
> > > > > > it is not.
> > > > > >
> > > > > > George
> > > > > >
> > > > > >
> > > > > >
> > > > > > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > > > > > Hi GZ,
> > > > > > >
> > > > > > > Can you try with "pData->parent" part applied too ?
> > > > > > > Sorry I should have mentioned this too.
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Nov 28, 2015 at 8:25 AM GZ <gz...@zygmund.net> wrote:
> > > > > > >
> > > > > > > > Hi Nobuaki,
> > > > > > > >
> > > > > > > > Yes - the rdata_write was in the generated c file not the
> > header.
> > > > > > > >
> > > > > > > > I made the changes you suggested, it was just what I was
> > looking
> > > > for. I
> > > > > > > > have repeated the corrected code to help others but
> > unfortunately
> > > > the
> > > > > > > > original problem persists..
> > > > > > > >
> > > > > > > >
> > > > > > > > tbuffer = g_object_new (THRIFT_TYPE_MEMORY_BUFFER,
> > "buf_size",
> > > > > > 1000,
> > > > > > > > NULL);
> > > > > > > > if(tbuffer){
> > > > > > > > transport = THRIFT_TRANSPORT(tbuffer);
> > > > > > > > thrift_transport_open (transport, &error);
> > > > > > > >
> > > > > > > > protocol = THRIFT_PROTOCOL(g_object_new
> > > > > > > > (THRIFT_TYPE_BINARY_PROTOCOL, "transport", transport, NULL));
> > > > > > > > if(protocol){
> > > > > > > > rdata* pData;
> > > > > > > > pData = g_object_new (TYPE_RDATA,"flag",1,
> > > > > > > > "id",27,
> > > > > > > > "start",12000,
> > > > > > > > "done",345435,
> > > > > > > > "notif",34455,
> > > > > > > >
> > > > > > > > "spid","XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",NULL);
> > > > > > > > if(pData){
> > > > > > > > ThriftStructClass* cls =
> > > > > > > > THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData));
> > > > > > > > len = cls->write(pData,protocol,&error);
> > > > > > > >
> > > > > > > > printf ("%d Bytes written to buffer\n", len);
> > > > > > > >
> > > > > > > > if(tbuffer->buf != NULL){
> > > > > > > > printHex((const void *)tbuffer->buf, 100);
> > > > > > > > }
> > > > > > > > g_object_unref (pData);
> > > > > > > > }
> > > > > > > > g_object_unref (protocol);
> > > > > > > > }
> > > > > > > > g_object_unref(tbuffer);
> > > > > > > > }
> > > > > > > >
> > > > > > > >
> > > > > > > > The returned buffer length still appears incorrect and the
> > string
> > > > does
> > > > > > not
> > > > > > > > appear to be serialised into the buffer.
> > > > > > > >
> > > > > > > > I can track the writes through the writing function and I
> > notice
> > > > that
> > > > > > for
> > > > > > > > the spid string value
> > > > > > > >
> > > > > > > > thrift_protocol_write_field_begin returns 3 bytes
> > > > > > > > thrift_protocol_write_string returns 50 bytes
> > > > > > > > thrift_protocol_write_field_end returns 0
> > > > > > > >
> > > > > > > > the 50 bytes never get added to the length because it is
> > overriden
> > > > by
> > > > > > the
> > > > > > > > 0. This is a snippit of the code..
> > > > > > > >
> > > > > > > > if ((ret = thrift_protocol_write_field_begin (protocol,
> > "spid",
> > > > > > > > T_STRING, 6, error)) < 0)
> > > > > > > > return -1;
> > > > > > > > xfer += ret;
> > > > > > > > if ((ret = thrift_protocol_write_string (protocol,
> > > > this_object->spid,
> > > > > > > > error)) < 0)
> > > > > > > > return -1;
> > > > > > > > if ((ret = thrift_protocol_write_field_end (protocol,
> > error)) <
> > > > 0)
> > > > > > > > return -1;
> > > > > > > > xfer += ret;
> > > > > > > >
> > > > > > > > The result is that the write function returns 19 bytes and
> > when I
> > > > view
> > > > > > the
> > > > > > > > buffer I see no string. I guess I must still be missing
> > > > something...
> > > > > > The
> > > > > > > > spid string is 46 characters long. I would expect to see this
> > > > > > reflected in
> > > > > > > > the length of the buffer.
> > > > > > > >
> > > > > > > > The idl record is
> > > > > > > >
> > > > > > > > enum Status {
> > > > > > > > ONE = 1,
> > > > > > > > TWO = 2,
> > > > > > > > THREE = 4,
> > > > > > > > }
> > > > > > > >
> > > > > > > > struct rdata {
> > > > > > > > 1: Status flag,
> > > > > > > > 2: i16 id,
> > > > > > > > 3: i64 start,
> > > > > > > > 4: i64 done,
> > > > > > > > 5: i64 notif,
> > > > > > > > 6: string spid
> > > > > > > > }
> > > > > > > >
> > > > > > > > Any idea what might be happenning?
> > > > > > > >
> > > > > > > > Thanks again...
> > > > > > > >
> > > > > > > > George
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > > > > > > > It is strange, the generated header has rdata_write ?
> > > > > > > > > Seeing generated code, it should be something like this:
> > > > > > > > >
> > > > > > > > > ThriftStructClass* cls =
> > > > THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData));
> > > > > > > > > cls->write(pData->parent, protocol, &error);
> > > > > > > > >
> > > > > > > > > On Thu, Nov 26, 2015 at 8:03 AM GZ <gz...@zygmund.net> wrote:
> > > > > > > > >
> > > > > > > > > > I am new to thrift and have been trying to serialize a
> > thrift
> > > > > > structure
> > > > > > > > > > using c but have been unable to get it to work. I want to
> > > > serialize
> > > > > > > > using
> > > > > > > > > > the binary protocol into a memory buffer. I do not need
> > RPC, I
> > > > > > will be
> > > > > > > > > > passing the message into Kafka. Does anyone know the
> > correct
> > > > way
> > > > > > to do
> > > > > > > > this?
> > > > > > > > > >
> > > > > > > > > > I have looked through all the examples and test code and
> > have
> > > > come
> > > > > > up
> > > > > > > > with
> > > > > > > > > > the following. It compiles, but is not working correctly.
> > I am
> > > > > > clearly
> > > > > > > > not
> > > > > > > > > > doing it correctly. rdata_write is a method generated by
> > > > thrift.
> > > > > > > > > >
> > > > > > > > > > GError *error = NULL;
> > > > > > > > > > ThriftTransport *transport = NULL;
> > > > > > > > > > ThriftProtocol *protocol = NULL;
> > > > > > > > > >
> > > > > > > > > > transport=g_object_new (THRIFT_TYPE_MEMORY_BUFFER,
> > > > "buf_size",
> > > > > > 100,
> > > > > > > > > > NULL);
> > > > > > > > > > if(transport){
> > > > > > > > > > thrift_transport_open (transport, &error);
> > > > > > > > > > protocol = g_object_new
> > (THRIFT_TYPE_BINARY_PROTOCOL,
> > > > > > > > "transport",
> > > > > > > > > > transport,NULL);
> > > > > > > > > > if(protocol){
> > > > > > > > > > rdata* pData = g_object_new
> > (TYPE_RDATA,"flag",1,
> > > > > > > > > > "id",27,
> > > > > > > > > > "start",12000,
> > > > > > > > > > "done",345435,
> > > > > > > > > > "notif",34455,
> > > > > > > > > > "spid","This is the
> > > > > > SPID",NULL);
> > > > > > > > > > if(pData){
> > > > > > > > > >
> > > > > > > > > > /* wanted something like
> > > > > > obj->write(protocol,&error)
> > > > > > > > > > But how to get object? pData does not
> > have
> > > > write
> > > > > > > > method
> > > > > > > > > > */
> > > > > > > > > >
> > > > > > > > > > gint32 len =
> > > > > > > > > > rdata_write(pData,(ThriftProtocol*)protocol,&error);
> > > > > > > > > >
> > > > > > > > > > // Print Content of Buffer
> > > > > > > > > > ThriftMemoryBuffer* pMem =
> > > > > > > > (ThriftMemoryBuffer*)transport;
> > > > > > > > > > if(pMem->buf != NULL){
> > > > > > > > > > printHex((const void *)pMem->buf, len);
> > > > > > > > > > }
> > > > > > > > > > g_object_unref (pData);
> > > > > > > > > > }
> > > > > > > > > > g_object_unref (protocol);
> > > > > > > > > > }
> > > > > > > > > > g_object_unref(transport);
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > Error handling and support functions skipped for brevity.
> > > > > > > > > >
> > > > > > > > > > Any help appreciated
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
Re: Serializing struct using c_glib into memory buffer Help!
Posted by Nobuaki Sukegawa <ns...@gmail.com>.
Hi GZ,
(const void *)tbuffer->buf seems problematic: it's a GByteArray*.
tbuffer->buf->data should point to the data.
On Sun, Dec 6, 2015 at 7:10 AM GZ <gz...@zygmund.net> wrote:
> Hi Nobuaki,
>
> That was my understanding too. So when I write the object I would expect
> to see the data serialised into the memory buffer... in tbuffer->buf But I
> dont. It would be hard to miss the long "XXXXX.." string.
>
> The data must be correctly stored because I was able load the data into a
> new object correctly.
>
> I did try your suggestion and try deserialise the buffer after it had been
> moved but this did not work, the object was empty.
>
> Its very frustrating. I tried the same process using the cpp library and
> it works!
>
> 013394F0 | 08000100 00000206 0002012C 0A000300 | ...........,....
> 01339500 | 00000000 63BF280A 00040000 00000063 | ....c.(........c
> 01339510 | C0F20A00 05000000 000063BF BE0B0006 | ..........c.....
> 01339520 | 0000001D 58585858 58585858 58585858 | ....XXXXXXXXXXXX
> 01339530 | 58585858 58585858 58585858 58585858 | XXXXXXXXXXXXXXXX
> 01339540 | 5800 | X.
>
> I think I must be getting the buffer incorrectly because the output above
> is completely different for the buffer I get in the c_glib version
> (identical object).
>
> Any Ideas ?!
>
> Thanks again
> George
>
>
>
> Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > Hi GZ,
> >
> > > string laid out as if it was going out "over-the-wire"
> > That's the whole point of memory buffer and it is supposed to work for
> your
> > use case.
> > Even with the correct length you don't see the string ?
> > You may want to just try sending the data of length reported by *read*
> > method and deserializing on the other side of the wire.
> >
> > On Tue, Dec 1, 2015 at 4:00 AM GZ <gz...@zygmund.net> wrote:
> >
> > > Hi Nobuaki,
> > >
> > > Thanks... I will keep an eye out for the fix. As you pointed out, using
> > > read returns the correct length (as far as I can tell) and I am able to
> > > deserialise all the data into a new object correctly. Thanks! However,
> when
> > > I look at the memory buffer, I still do not see the string. I suspect
> that
> > > the buffer contains a pointer to the string, not the string laid out
> as if
> > > it was going out "over-the-wire".
> > >
> > > Mayber I am accessing the buffer incorrectly or using the wrong
> transport?
> > > Ultimately I want to be able to take the whole buffer and send it to
> Kafka
> > > as a message for desrialisation by other components.
> > >
> > > Your help is much appreciated!
> > >
> > > George
> > >
> > >
> > >
> > >
> > > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > > Hi GZ,
> > > >
> > > > You're absolutely right about both "&" and the length/contents.
> > > > It turned out that it is a bug: as you pointed out the write method
> > > ignores
> > > > some bytes written.
> > > > So the data is written and ready to be deserialized but the reported
> > > length
> > > > is wrong.
> > > > If you happen to be building Thrift from the source, the fix will be
> > > > available on master branch soon.
> > > >
> > > > A very dirty and non-performant workaround is to deserialize the
> buffer
> > > > only to know the byte length before sending:
> > > > wrong_len = cls->write(THRIFT_STRUCT(pData2), ...);
> > > > correct_len = cls->read(THRIFT_STRUCT(pData2), ...);
> > > > you'll see correct byte length since read is not affected by this
> bug.
> > > >
> > > > On Sun, Nov 29, 2015 at 4:22 AM GZ <gz...@zygmund.net> wrote:
> > > >
> > > > > Hi Nobuaki,
> > > > >
> > > > > I tried this originally but write is expecting a ThriftStruct*
> rather
> > > than
> > > > > a ThriftStruct. I have tried with both &pData->parent and pData..
> the
> > > > > results are pretty much the same. Below is a Hex printout of the
> memory
> > > > > buffer (19 bytes).
> > > > >
> > > > > &pData->parent
> > > > > 01FB5040 | 1081FB01 00000000 52000000 80000000 | ........R.......
> > > > > 01FB5050 | 010000
> > > > >
> > > > > pData
> > > > > 01EA4040 | 1071EA01 00000000 52000000 80000000 | .q......R.......
> > > > > 01EA4050 | 010000
> > > > >
> > > > >
> > > > > I would expect to see the "XXXXXXX" string serialised into this
> buffer
> > > but
> > > > > it is not.
> > > > >
> > > > > George
> > > > >
> > > > >
> > > > >
> > > > > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > > > > Hi GZ,
> > > > > >
> > > > > > Can you try with "pData->parent" part applied too ?
> > > > > > Sorry I should have mentioned this too.
> > > > > >
> > > > > >
> > > > > > On Sat, Nov 28, 2015 at 8:25 AM GZ <gz...@zygmund.net> wrote:
> > > > > >
> > > > > > > Hi Nobuaki,
> > > > > > >
> > > > > > > Yes - the rdata_write was in the generated c file not the
> header.
> > > > > > >
> > > > > > > I made the changes you suggested, it was just what I was
> looking
> > > for. I
> > > > > > > have repeated the corrected code to help others but
> unfortunately
> > > the
> > > > > > > original problem persists..
> > > > > > >
> > > > > > >
> > > > > > > tbuffer = g_object_new (THRIFT_TYPE_MEMORY_BUFFER,
> "buf_size",
> > > > > 1000,
> > > > > > > NULL);
> > > > > > > if(tbuffer){
> > > > > > > transport = THRIFT_TRANSPORT(tbuffer);
> > > > > > > thrift_transport_open (transport, &error);
> > > > > > >
> > > > > > > protocol = THRIFT_PROTOCOL(g_object_new
> > > > > > > (THRIFT_TYPE_BINARY_PROTOCOL, "transport", transport, NULL));
> > > > > > > if(protocol){
> > > > > > > rdata* pData;
> > > > > > > pData = g_object_new (TYPE_RDATA,"flag",1,
> > > > > > > "id",27,
> > > > > > > "start",12000,
> > > > > > > "done",345435,
> > > > > > > "notif",34455,
> > > > > > >
> > > > > > > "spid","XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",NULL);
> > > > > > > if(pData){
> > > > > > > ThriftStructClass* cls =
> > > > > > > THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData));
> > > > > > > len = cls->write(pData,protocol,&error);
> > > > > > >
> > > > > > > printf ("%d Bytes written to buffer\n", len);
> > > > > > >
> > > > > > > if(tbuffer->buf != NULL){
> > > > > > > printHex((const void *)tbuffer->buf, 100);
> > > > > > > }
> > > > > > > g_object_unref (pData);
> > > > > > > }
> > > > > > > g_object_unref (protocol);
> > > > > > > }
> > > > > > > g_object_unref(tbuffer);
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > > The returned buffer length still appears incorrect and the
> string
> > > does
> > > > > not
> > > > > > > appear to be serialised into the buffer.
> > > > > > >
> > > > > > > I can track the writes through the writing function and I
> notice
> > > that
> > > > > for
> > > > > > > the spid string value
> > > > > > >
> > > > > > > thrift_protocol_write_field_begin returns 3 bytes
> > > > > > > thrift_protocol_write_string returns 50 bytes
> > > > > > > thrift_protocol_write_field_end returns 0
> > > > > > >
> > > > > > > the 50 bytes never get added to the length because it is
> overriden
> > > by
> > > > > the
> > > > > > > 0. This is a snippit of the code..
> > > > > > >
> > > > > > > if ((ret = thrift_protocol_write_field_begin (protocol,
> "spid",
> > > > > > > T_STRING, 6, error)) < 0)
> > > > > > > return -1;
> > > > > > > xfer += ret;
> > > > > > > if ((ret = thrift_protocol_write_string (protocol,
> > > this_object->spid,
> > > > > > > error)) < 0)
> > > > > > > return -1;
> > > > > > > if ((ret = thrift_protocol_write_field_end (protocol,
> error)) <
> > > 0)
> > > > > > > return -1;
> > > > > > > xfer += ret;
> > > > > > >
> > > > > > > The result is that the write function returns 19 bytes and
> when I
> > > view
> > > > > the
> > > > > > > buffer I see no string. I guess I must still be missing
> > > something...
> > > > > The
> > > > > > > spid string is 46 characters long. I would expect to see this
> > > > > reflected in
> > > > > > > the length of the buffer.
> > > > > > >
> > > > > > > The idl record is
> > > > > > >
> > > > > > > enum Status {
> > > > > > > ONE = 1,
> > > > > > > TWO = 2,
> > > > > > > THREE = 4,
> > > > > > > }
> > > > > > >
> > > > > > > struct rdata {
> > > > > > > 1: Status flag,
> > > > > > > 2: i16 id,
> > > > > > > 3: i64 start,
> > > > > > > 4: i64 done,
> > > > > > > 5: i64 notif,
> > > > > > > 6: string spid
> > > > > > > }
> > > > > > >
> > > > > > > Any idea what might be happenning?
> > > > > > >
> > > > > > > Thanks again...
> > > > > > >
> > > > > > > George
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > > > > > > It is strange, the generated header has rdata_write ?
> > > > > > > > Seeing generated code, it should be something like this:
> > > > > > > >
> > > > > > > > ThriftStructClass* cls =
> > > THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData));
> > > > > > > > cls->write(pData->parent, protocol, &error);
> > > > > > > >
> > > > > > > > On Thu, Nov 26, 2015 at 8:03 AM GZ <gz...@zygmund.net> wrote:
> > > > > > > >
> > > > > > > > > I am new to thrift and have been trying to serialize a
> thrift
> > > > > structure
> > > > > > > > > using c but have been unable to get it to work. I want to
> > > serialize
> > > > > > > using
> > > > > > > > > the binary protocol into a memory buffer. I do not need
> RPC, I
> > > > > will be
> > > > > > > > > passing the message into Kafka. Does anyone know the
> correct
> > > way
> > > > > to do
> > > > > > > this?
> > > > > > > > >
> > > > > > > > > I have looked through all the examples and test code and
> have
> > > come
> > > > > up
> > > > > > > with
> > > > > > > > > the following. It compiles, but is not working correctly.
> I am
> > > > > clearly
> > > > > > > not
> > > > > > > > > doing it correctly. rdata_write is a method generated by
> > > thrift.
> > > > > > > > >
> > > > > > > > > GError *error = NULL;
> > > > > > > > > ThriftTransport *transport = NULL;
> > > > > > > > > ThriftProtocol *protocol = NULL;
> > > > > > > > >
> > > > > > > > > transport=g_object_new (THRIFT_TYPE_MEMORY_BUFFER,
> > > "buf_size",
> > > > > 100,
> > > > > > > > > NULL);
> > > > > > > > > if(transport){
> > > > > > > > > thrift_transport_open (transport, &error);
> > > > > > > > > protocol = g_object_new
> (THRIFT_TYPE_BINARY_PROTOCOL,
> > > > > > > "transport",
> > > > > > > > > transport,NULL);
> > > > > > > > > if(protocol){
> > > > > > > > > rdata* pData = g_object_new
> (TYPE_RDATA,"flag",1,
> > > > > > > > > "id",27,
> > > > > > > > > "start",12000,
> > > > > > > > > "done",345435,
> > > > > > > > > "notif",34455,
> > > > > > > > > "spid","This is the
> > > > > SPID",NULL);
> > > > > > > > > if(pData){
> > > > > > > > >
> > > > > > > > > /* wanted something like
> > > > > obj->write(protocol,&error)
> > > > > > > > > But how to get object? pData does not
> have
> > > write
> > > > > > > method
> > > > > > > > > */
> > > > > > > > >
> > > > > > > > > gint32 len =
> > > > > > > > > rdata_write(pData,(ThriftProtocol*)protocol,&error);
> > > > > > > > >
> > > > > > > > > // Print Content of Buffer
> > > > > > > > > ThriftMemoryBuffer* pMem =
> > > > > > > (ThriftMemoryBuffer*)transport;
> > > > > > > > > if(pMem->buf != NULL){
> > > > > > > > > printHex((const void *)pMem->buf, len);
> > > > > > > > > }
> > > > > > > > > g_object_unref (pData);
> > > > > > > > > }
> > > > > > > > > g_object_unref (protocol);
> > > > > > > > > }
> > > > > > > > > g_object_unref(transport);
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > Error handling and support functions skipped for brevity.
> > > > > > > > >
> > > > > > > > > Any help appreciated
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>
Re: Serializing struct using c_glib into memory buffer Help!
Posted by GZ <gz...@zygmund.net>.
Hi Nobuaki,
That was my understanding too. So when I write the object I would expect to see the data serialised into the memory buffer... in tbuffer->buf But I dont. It would be hard to miss the long "XXXXX.." string.
The data must be correctly stored because I was able load the data into a new object correctly.
I did try your suggestion and try deserialise the buffer after it had been moved but this did not work, the object was empty.
Its very frustrating. I tried the same process using the cpp library and it works!
013394F0 | 08000100 00000206 0002012C 0A000300 | ...........,....
01339500 | 00000000 63BF280A 00040000 00000063 | ....c.(........c
01339510 | C0F20A00 05000000 000063BF BE0B0006 | ..........c.....
01339520 | 0000001D 58585858 58585858 58585858 | ....XXXXXXXXXXXX
01339530 | 58585858 58585858 58585858 58585858 | XXXXXXXXXXXXXXXX
01339540 | 5800 | X.
I think I must be getting the buffer incorrectly because the output above is completely different for the buffer I get in the c_glib version (identical object).
Any Ideas ?!
Thanks again
George
Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> Hi GZ,
>
> > string laid out as if it was going out "over-the-wire"
> That's the whole point of memory buffer and it is supposed to work for your
> use case.
> Even with the correct length you don't see the string ?
> You may want to just try sending the data of length reported by *read*
> method and deserializing on the other side of the wire.
>
> On Tue, Dec 1, 2015 at 4:00 AM GZ <gz...@zygmund.net> wrote:
>
> > Hi Nobuaki,
> >
> > Thanks... I will keep an eye out for the fix. As you pointed out, using
> > read returns the correct length (as far as I can tell) and I am able to
> > deserialise all the data into a new object correctly. Thanks! However, when
> > I look at the memory buffer, I still do not see the string. I suspect that
> > the buffer contains a pointer to the string, not the string laid out as if
> > it was going out "over-the-wire".
> >
> > Mayber I am accessing the buffer incorrectly or using the wrong transport?
> > Ultimately I want to be able to take the whole buffer and send it to Kafka
> > as a message for desrialisation by other components.
> >
> > Your help is much appreciated!
> >
> > George
> >
> >
> >
> >
> > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > Hi GZ,
> > >
> > > You're absolutely right about both "&" and the length/contents.
> > > It turned out that it is a bug: as you pointed out the write method
> > ignores
> > > some bytes written.
> > > So the data is written and ready to be deserialized but the reported
> > length
> > > is wrong.
> > > If you happen to be building Thrift from the source, the fix will be
> > > available on master branch soon.
> > >
> > > A very dirty and non-performant workaround is to deserialize the buffer
> > > only to know the byte length before sending:
> > > wrong_len = cls->write(THRIFT_STRUCT(pData2), ...);
> > > correct_len = cls->read(THRIFT_STRUCT(pData2), ...);
> > > you'll see correct byte length since read is not affected by this bug.
> > >
> > > On Sun, Nov 29, 2015 at 4:22 AM GZ <gz...@zygmund.net> wrote:
> > >
> > > > Hi Nobuaki,
> > > >
> > > > I tried this originally but write is expecting a ThriftStruct* rather
> > than
> > > > a ThriftStruct. I have tried with both &pData->parent and pData.. the
> > > > results are pretty much the same. Below is a Hex printout of the memory
> > > > buffer (19 bytes).
> > > >
> > > > &pData->parent
> > > > 01FB5040 | 1081FB01 00000000 52000000 80000000 | ........R.......
> > > > 01FB5050 | 010000
> > > >
> > > > pData
> > > > 01EA4040 | 1071EA01 00000000 52000000 80000000 | .q......R.......
> > > > 01EA4050 | 010000
> > > >
> > > >
> > > > I would expect to see the "XXXXXXX" string serialised into this buffer
> > but
> > > > it is not.
> > > >
> > > > George
> > > >
> > > >
> > > >
> > > > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > > > Hi GZ,
> > > > >
> > > > > Can you try with "pData->parent" part applied too ?
> > > > > Sorry I should have mentioned this too.
> > > > >
> > > > >
> > > > > On Sat, Nov 28, 2015 at 8:25 AM GZ <gz...@zygmund.net> wrote:
> > > > >
> > > > > > Hi Nobuaki,
> > > > > >
> > > > > > Yes - the rdata_write was in the generated c file not the header.
> > > > > >
> > > > > > I made the changes you suggested, it was just what I was looking
> > for. I
> > > > > > have repeated the corrected code to help others but unfortunately
> > the
> > > > > > original problem persists..
> > > > > >
> > > > > >
> > > > > > tbuffer = g_object_new (THRIFT_TYPE_MEMORY_BUFFER, "buf_size",
> > > > 1000,
> > > > > > NULL);
> > > > > > if(tbuffer){
> > > > > > transport = THRIFT_TRANSPORT(tbuffer);
> > > > > > thrift_transport_open (transport, &error);
> > > > > >
> > > > > > protocol = THRIFT_PROTOCOL(g_object_new
> > > > > > (THRIFT_TYPE_BINARY_PROTOCOL, "transport", transport, NULL));
> > > > > > if(protocol){
> > > > > > rdata* pData;
> > > > > > pData = g_object_new (TYPE_RDATA,"flag",1,
> > > > > > "id",27,
> > > > > > "start",12000,
> > > > > > "done",345435,
> > > > > > "notif",34455,
> > > > > >
> > > > > > "spid","XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",NULL);
> > > > > > if(pData){
> > > > > > ThriftStructClass* cls =
> > > > > > THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData));
> > > > > > len = cls->write(pData,protocol,&error);
> > > > > >
> > > > > > printf ("%d Bytes written to buffer\n", len);
> > > > > >
> > > > > > if(tbuffer->buf != NULL){
> > > > > > printHex((const void *)tbuffer->buf, 100);
> > > > > > }
> > > > > > g_object_unref (pData);
> > > > > > }
> > > > > > g_object_unref (protocol);
> > > > > > }
> > > > > > g_object_unref(tbuffer);
> > > > > > }
> > > > > >
> > > > > >
> > > > > > The returned buffer length still appears incorrect and the string
> > does
> > > > not
> > > > > > appear to be serialised into the buffer.
> > > > > >
> > > > > > I can track the writes through the writing function and I notice
> > that
> > > > for
> > > > > > the spid string value
> > > > > >
> > > > > > thrift_protocol_write_field_begin returns 3 bytes
> > > > > > thrift_protocol_write_string returns 50 bytes
> > > > > > thrift_protocol_write_field_end returns 0
> > > > > >
> > > > > > the 50 bytes never get added to the length because it is overriden
> > by
> > > > the
> > > > > > 0. This is a snippit of the code..
> > > > > >
> > > > > > if ((ret = thrift_protocol_write_field_begin (protocol, "spid",
> > > > > > T_STRING, 6, error)) < 0)
> > > > > > return -1;
> > > > > > xfer += ret;
> > > > > > if ((ret = thrift_protocol_write_string (protocol,
> > this_object->spid,
> > > > > > error)) < 0)
> > > > > > return -1;
> > > > > > if ((ret = thrift_protocol_write_field_end (protocol, error)) <
> > 0)
> > > > > > return -1;
> > > > > > xfer += ret;
> > > > > >
> > > > > > The result is that the write function returns 19 bytes and when I
> > view
> > > > the
> > > > > > buffer I see no string. I guess I must still be missing
> > something...
> > > > The
> > > > > > spid string is 46 characters long. I would expect to see this
> > > > reflected in
> > > > > > the length of the buffer.
> > > > > >
> > > > > > The idl record is
> > > > > >
> > > > > > enum Status {
> > > > > > ONE = 1,
> > > > > > TWO = 2,
> > > > > > THREE = 4,
> > > > > > }
> > > > > >
> > > > > > struct rdata {
> > > > > > 1: Status flag,
> > > > > > 2: i16 id,
> > > > > > 3: i64 start,
> > > > > > 4: i64 done,
> > > > > > 5: i64 notif,
> > > > > > 6: string spid
> > > > > > }
> > > > > >
> > > > > > Any idea what might be happenning?
> > > > > >
> > > > > > Thanks again...
> > > > > >
> > > > > > George
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Nobuaki Sukegawa <ns...@gmail.com> wrote ..
> > > > > > > It is strange, the generated header has rdata_write ?
> > > > > > > Seeing generated code, it should be something like this:
> > > > > > >
> > > > > > > ThriftStructClass* cls =
> > THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData));
> > > > > > > cls->write(pData->parent, protocol, &error);
> > > > > > >
> > > > > > > On Thu, Nov 26, 2015 at 8:03 AM GZ <gz...@zygmund.net> wrote:
> > > > > > >
> > > > > > > > I am new to thrift and have been trying to serialize a thrift
> > > > structure
> > > > > > > > using c but have been unable to get it to work. I want to
> > serialize
> > > > > > using
> > > > > > > > the binary protocol into a memory buffer. I do not need RPC, I
> > > > will be
> > > > > > > > passing the message into Kafka. Does anyone know the correct
> > way
> > > > to do
> > > > > > this?
> > > > > > > >
> > > > > > > > I have looked through all the examples and test code and have
> > come
> > > > up
> > > > > > with
> > > > > > > > the following. It compiles, but is not working correctly. I am
> > > > clearly
> > > > > > not
> > > > > > > > doing it correctly. rdata_write is a method generated by
> > thrift.
> > > > > > > >
> > > > > > > > GError *error = NULL;
> > > > > > > > ThriftTransport *transport = NULL;
> > > > > > > > ThriftProtocol *protocol = NULL;
> > > > > > > >
> > > > > > > > transport=g_object_new (THRIFT_TYPE_MEMORY_BUFFER,
> > "buf_size",
> > > > 100,
> > > > > > > > NULL);
> > > > > > > > if(transport){
> > > > > > > > thrift_transport_open (transport, &error);
> > > > > > > > protocol = g_object_new (THRIFT_TYPE_BINARY_PROTOCOL,
> > > > > > "transport",
> > > > > > > > transport,NULL);
> > > > > > > > if(protocol){
> > > > > > > > rdata* pData = g_object_new (TYPE_RDATA,"flag",1,
> > > > > > > > "id",27,
> > > > > > > > "start",12000,
> > > > > > > > "done",345435,
> > > > > > > > "notif",34455,
> > > > > > > > "spid","This is the
> > > > SPID",NULL);
> > > > > > > > if(pData){
> > > > > > > >
> > > > > > > > /* wanted something like
> > > > obj->write(protocol,&error)
> > > > > > > > But how to get object? pData does not have
> > write
> > > > > > method
> > > > > > > > */
> > > > > > > >
> > > > > > > > gint32 len =
> > > > > > > > rdata_write(pData,(ThriftProtocol*)protocol,&error);
> > > > > > > >
> > > > > > > > // Print Content of Buffer
> > > > > > > > ThriftMemoryBuffer* pMem =
> > > > > > (ThriftMemoryBuffer*)transport;
> > > > > > > > if(pMem->buf != NULL){
> > > > > > > > printHex((const void *)pMem->buf, len);
> > > > > > > > }
> > > > > > > > g_object_unref (pData);
> > > > > > > > }
> > > > > > > > g_object_unref (protocol);
> > > > > > > > }
> > > > > > > > g_object_unref(transport);
> > > > > > > > }
> > > > > > > >
> > > > > > > > Error handling and support functions skipped for brevity.
> > > > > > > >
> > > > > > > > Any help appreciated
> > > > > > > >
> > > > > >
> > > >
> >