You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@thrift.apache.org by Rodolfo Ochoa <ro...@gmail.com> on 2014/10/15 20:07:19 UTC
Re: Error with binary data with node.js
where is this generated?
how can I generate my own?
On Sat, Sep 20, 2014 at 6:03 PM, Randy Abernethy <ra...@gmail.com>
wrote:
> I think the official answer to that is always "real soon now".
>
> In the mean time you can get the current master with npm as follows:
>
> $ npm install RandyAbernethy/thriftnode
>
> This is a clone of the thrift master branch setup to work with npm.
>
>
> On Fri, Sep 19, 2014 at 4:41 PM, Rodolfo Ochoa <ro...@gmail.com>
> wrote:
> > Oh, it is fixed on newest version...
> >
> > do you have a release date?
> >
> > thanks
> >
> > On Fri, Sep 19, 2014 at 6:24 PM, Rodolfo Ochoa <ro...@gmail.com>
> > wrote:
> >
> >> this error start happening here:
> >> https://issues.apache.org/jira/browse/THRIFT-1679
> >> which is a correct fix, but then you need to correctly implement binary.
> >>
> >> -Rod.O
> >>
> >> On Fri, Sep 19, 2014 at 5:33 PM, Rodolfo Ochoa <rodolfo.ochoa@gmail.com
> >
> >> wrote:
> >>
> >>> the problem is...
> >>> the problem is that for the generated code,
> >>> thrift maps binary data into strings, and uses the method readString
> from
> >>> the BinaryProtocol
> >>> protocol.js:280 TBinaryProtocol.prototype.readString = function() {
> >>> instead of using readBinary,
> >>> protocol.js:275 TBinaryProtocol.prototype.readBinary = function() {
> >>> which would return a buffer object.
> >>> readString method in the transport will use
> buffer(...).toString("utf8")
> >>> which will sanitize utf and will replace invalid unicode sequences
> with the
> >>> Unicode Replacement Character.
> >>>
> >>> *my code:*
> >>> my binary field was mapped on ondb_types.js as:
> >>> TGetResult.prototype = {};
> >>> TGetResult.prototype.read = function(input) {
> >>> input.readStructBegin();
> >>> while (true)
> >>> {
> >>> var ret = input.readFieldBegin();
> >>> var fname = ret.fname;
> >>> var ftype = ret.ftype;
> >>> var fid = ret.fid;
> >>> if (ftype == Thrift.Type.STOP) {
> >>> break;
> >>> }
> >>> switch (fid)
> >>> {
> >>> case 1:
> >>> if (ftype == Thrift.Type.STRUCT) {
> >>> this.currentRow = new ttypes.TRow();
> >>> this.currentRow.read(input);
> >>> } else {
> >>> input.skip(ftype);
> >>> }
> >>> break;
> >>> case 2:
> >>> * if (ftype == Thrift.Type.STRING) {*
> >>> * this.currentRowVersion = input.readString();*
> >>> } else {
> >>> input.skip(ftype);
> >>> }
> >>> break;
> >>> default:
> >>> input.skip(ftype);
> >>> }
> >>> input.readFieldEnd();
> >>> }
> >>> input.readStructEnd();
> >>> return;
> >>> };
> >>>
> >>>
> >>> and TGetResult is defined in ondb.thrift file as:
> >>> typedef binary TVersion
> >>> struct TGetResult
> >>> {
> >>> 1:TRow currentRow;
> >>> 2:TVersion currentRowVersion;
> >>> }
> >>>
> >>> I'm not sure if this is already been reported or if this is already
> been
> >>> fixed on the newest version.
> >>>
> >>> -Rod.O
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Sep 18, 2014 at 11:23 PM, Randy Abernethy <
> >>> randy.abernethy@gmail.com> wrote:
> >>>
> >>>> Hey Rodolfo,
> >>>>
> >>>> While I can not say for sure not having seen your code, I would be
> highly
> >>>> suspect of any Thift node lib prior to 0.9.2, which is not yet
> released.
> >>>> Try the current master, it has too many fixes to mention and will
> likely
> >>>> solve your problem. You don't need to change versions on any of the
> other
> >>>> languages. Ping back if you still have trouble.
> >>>>
> >>>> Best,
> >>>> Randy
> >>>>
> >>>> On Thursday, September 18, 2014, Rodolfo Ochoa <
> rodolfo.ochoa@gmail.com>
> >>>> wrote:
> >>>>
> >>>> > on node:
> >>>> > thrift@0.9.1
> >>>> >
> >>>> > on java:
> >>>> > libthrift-0.9.1.jar
> >>>> >
> >>>> >
> >>>> >
> >>>> > On Thu, Sep 18, 2014 at 9:09 PM, Randy Abernethy <
> >>>> > randy.abernethy@gmail.com <javascript:;>>
> >>>> > wrote:
> >>>> >
> >>>> > > What version of thrift are you using?
> >>>> > > On Sep 18, 2014 7:46 PM, "Rodolfo Ochoa" <rodolfo.ochoa@gmail.com
> >>>> > <javascript:;>> wrote:
> >>>> > >
> >>>> > > > I have a Java server sending info to C/Python/node.js clients,
> >>>> > > > I don't have issues with C or Python, only with node.js
> >>>> > > > on Java side, I'm sending a byte array, and if I print the
> values
> >>>> of
> >>>> > this
> >>>> > > > byte array I got:
> >>>> > > > byte codes: -84 -19 0 5 119 44 0 5 -50 108 0 44 75 73 78 -120
> >>>> -103 -59
> >>>> > > > -124 3 -92 -121 -98 -104 0 0 0 0 0 0 37 -128 1 3 0 0 0 1 0 0 0
> 1 0
> >>>> 0 0
> >>>> > 0
> >>>> > > 0
> >>>> > > > -78 68 -89
> >>>> > > >
> >>>> > > > but on node.js side I receive:
> >>>> > > > byte codes: 65533 65533 0 5 119 44 0 5 65533 108 0 44 75 73 78
> >>>> 65533
> >>>> > > 65533
> >>>> > > > 324 3 65533 65533 65533 65533 0 0 0 0 0 0 37 65533 1 3 0 0 0 1
> 0 0
> >>>> 0 1
> >>>> > 0
> >>>> > > 0
> >>>> > > > 0 0 0 65533 68 65533
> >>>> > > >
> >>>> > > > this means that some characters were replaced, why? how?
> >>>> > > > how can avoid this behavior?
> >>>> > > >
> >>>> > > >
> >>>> > > >
> >>>> > > > --
> >>>> > > > *Rod.O*
> >>>> > > >
> >>>> > >
> >>>> >
> >>>> >
> >>>> >
> >>>> > --
> >>>> > *Rod.O*
> >>>> >
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> *Rod.O*
> >>>
> >>
> >>
> >>
> >> --
> >> *Rod.O*
> >>
> >
> >
> >
> > --
> > *Rod.O*
>
--
*Rod.O*
Re: Error with binary data with node.js
Posted by Randy Abernethy <ra...@gmail.com>.
The source at RandyAbernethy/thriftnode is a copy of
https://github.com/apache/thrift/tree/master/lib/nodejs. You can
easily make your own copy but you can not just fork the apache/thrift
repo (which is why I maintain a node only repo). The semver tags on
the apache repo will cause npm to use code from over a year ago (which
has lots of issues). This will be fixed when 0.9.2 is released in the
near future, causing npm install to pull a well tested node source
base.
On Wed, Oct 15, 2014 at 11:07 AM, Rodolfo Ochoa <ro...@gmail.com> wrote:
> where is this generated?
> how can I generate my own?
>
> On Sat, Sep 20, 2014 at 6:03 PM, Randy Abernethy <ra...@gmail.com>
> wrote:
>
>> I think the official answer to that is always "real soon now".
>>
>> In the mean time you can get the current master with npm as follows:
>>
>> $ npm install RandyAbernethy/thriftnode
>>
>> This is a clone of the thrift master branch setup to work with npm.
>>
>>
>> On Fri, Sep 19, 2014 at 4:41 PM, Rodolfo Ochoa <ro...@gmail.com>
>> wrote:
>> > Oh, it is fixed on newest version...
>> >
>> > do you have a release date?
>> >
>> > thanks
>> >
>> > On Fri, Sep 19, 2014 at 6:24 PM, Rodolfo Ochoa <ro...@gmail.com>
>> > wrote:
>> >
>> >> this error start happening here:
>> >> https://issues.apache.org/jira/browse/THRIFT-1679
>> >> which is a correct fix, but then you need to correctly implement binary.
>> >>
>> >> -Rod.O
>> >>
>> >> On Fri, Sep 19, 2014 at 5:33 PM, Rodolfo Ochoa <rodolfo.ochoa@gmail.com
>> >
>> >> wrote:
>> >>
>> >>> the problem is...
>> >>> the problem is that for the generated code,
>> >>> thrift maps binary data into strings, and uses the method readString
>> from
>> >>> the BinaryProtocol
>> >>> protocol.js:280 TBinaryProtocol.prototype.readString = function() {
>> >>> instead of using readBinary,
>> >>> protocol.js:275 TBinaryProtocol.prototype.readBinary = function() {
>> >>> which would return a buffer object.
>> >>> readString method in the transport will use
>> buffer(...).toString("utf8")
>> >>> which will sanitize utf and will replace invalid unicode sequences
>> with the
>> >>> Unicode Replacement Character.
>> >>>
>> >>> *my code:*
>> >>> my binary field was mapped on ondb_types.js as:
>> >>> TGetResult.prototype = {};
>> >>> TGetResult.prototype.read = function(input) {
>> >>> input.readStructBegin();
>> >>> while (true)
>> >>> {
>> >>> var ret = input.readFieldBegin();
>> >>> var fname = ret.fname;
>> >>> var ftype = ret.ftype;
>> >>> var fid = ret.fid;
>> >>> if (ftype == Thrift.Type.STOP) {
>> >>> break;
>> >>> }
>> >>> switch (fid)
>> >>> {
>> >>> case 1:
>> >>> if (ftype == Thrift.Type.STRUCT) {
>> >>> this.currentRow = new ttypes.TRow();
>> >>> this.currentRow.read(input);
>> >>> } else {
>> >>> input.skip(ftype);
>> >>> }
>> >>> break;
>> >>> case 2:
>> >>> * if (ftype == Thrift.Type.STRING) {*
>> >>> * this.currentRowVersion = input.readString();*
>> >>> } else {
>> >>> input.skip(ftype);
>> >>> }
>> >>> break;
>> >>> default:
>> >>> input.skip(ftype);
>> >>> }
>> >>> input.readFieldEnd();
>> >>> }
>> >>> input.readStructEnd();
>> >>> return;
>> >>> };
>> >>>
>> >>>
>> >>> and TGetResult is defined in ondb.thrift file as:
>> >>> typedef binary TVersion
>> >>> struct TGetResult
>> >>> {
>> >>> 1:TRow currentRow;
>> >>> 2:TVersion currentRowVersion;
>> >>> }
>> >>>
>> >>> I'm not sure if this is already been reported or if this is already
>> been
>> >>> fixed on the newest version.
>> >>>
>> >>> -Rod.O
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Sep 18, 2014 at 11:23 PM, Randy Abernethy <
>> >>> randy.abernethy@gmail.com> wrote:
>> >>>
>> >>>> Hey Rodolfo,
>> >>>>
>> >>>> While I can not say for sure not having seen your code, I would be
>> highly
>> >>>> suspect of any Thift node lib prior to 0.9.2, which is not yet
>> released.
>> >>>> Try the current master, it has too many fixes to mention and will
>> likely
>> >>>> solve your problem. You don't need to change versions on any of the
>> other
>> >>>> languages. Ping back if you still have trouble.
>> >>>>
>> >>>> Best,
>> >>>> Randy
>> >>>>
>> >>>> On Thursday, September 18, 2014, Rodolfo Ochoa <
>> rodolfo.ochoa@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> > on node:
>> >>>> > thrift@0.9.1
>> >>>> >
>> >>>> > on java:
>> >>>> > libthrift-0.9.1.jar
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > On Thu, Sep 18, 2014 at 9:09 PM, Randy Abernethy <
>> >>>> > randy.abernethy@gmail.com <javascript:;>>
>> >>>> > wrote:
>> >>>> >
>> >>>> > > What version of thrift are you using?
>> >>>> > > On Sep 18, 2014 7:46 PM, "Rodolfo Ochoa" <rodolfo.ochoa@gmail.com
>> >>>> > <javascript:;>> wrote:
>> >>>> > >
>> >>>> > > > I have a Java server sending info to C/Python/node.js clients,
>> >>>> > > > I don't have issues with C or Python, only with node.js
>> >>>> > > > on Java side, I'm sending a byte array, and if I print the
>> values
>> >>>> of
>> >>>> > this
>> >>>> > > > byte array I got:
>> >>>> > > > byte codes: -84 -19 0 5 119 44 0 5 -50 108 0 44 75 73 78 -120
>> >>>> -103 -59
>> >>>> > > > -124 3 -92 -121 -98 -104 0 0 0 0 0 0 37 -128 1 3 0 0 0 1 0 0 0
>> 1 0
>> >>>> 0 0
>> >>>> > 0
>> >>>> > > 0
>> >>>> > > > -78 68 -89
>> >>>> > > >
>> >>>> > > > but on node.js side I receive:
>> >>>> > > > byte codes: 65533 65533 0 5 119 44 0 5 65533 108 0 44 75 73 78
>> >>>> 65533
>> >>>> > > 65533
>> >>>> > > > 324 3 65533 65533 65533 65533 0 0 0 0 0 0 37 65533 1 3 0 0 0 1
>> 0 0
>> >>>> 0 1
>> >>>> > 0
>> >>>> > > 0
>> >>>> > > > 0 0 0 65533 68 65533
>> >>>> > > >
>> >>>> > > > this means that some characters were replaced, why? how?
>> >>>> > > > how can avoid this behavior?
>> >>>> > > >
>> >>>> > > >
>> >>>> > > >
>> >>>> > > > --
>> >>>> > > > *Rod.O*
>> >>>> > > >
>> >>>> > >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > --
>> >>>> > *Rod.O*
>> >>>> >
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> *Rod.O*
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> *Rod.O*
>> >>
>> >
>> >
>> >
>> > --
>> > *Rod.O*
>>
>
>
>
> --
> *Rod.O*