You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Sam Peka <sa...@gmail.com> on 2020/10/28 13:39:46 UTC
[JS] BigNum toJSON returns a string with quotations in it
Hi there,
The toJSON method of BigNum is currently returning a string with quotation marks around it: https://github.com/apache/arrow/blob/fbb781bcdd15d54c52c71881c5c53d0c68069be6/js/src/util/bn.ts#L39
JSON.stringify will include those quotation marks in the output JSON. Here’s an example illustrating the issue:
const a = BN.new([12])
console.log(JSON.stringify({ a }))
> {"a":"\"12\""}
Our usecase is to take some arrow data and return it to a web client as json – but because the format is bad we’re having to work around this by converting all BigNums to strings before serilaizing the response.
Is BigNum.toJSON intended to work with JSON.stringify? Happy to open a PR if so.
Best,
Sam
Re: [JS] BigNum toJSON returns a string with quotations in it
Posted by Paul Taylor <pt...@gmail.com>.
The JSON rep will still have quotes when returned by JSON.stringify(), by virtue of being a string. The current code is causing the decimal to be double-quoted, with a set of escaped inner quotes.
> On Oct 28, 2020, at 10:12 AM, Lee, David <Da...@blackrock.com.invalid> wrote:
>
> Quotes shouldn't be removed. BigNum is a decimal derivative and decimals in JSON have quotes.
>
> This is needed to distinguish values like "2.1" vs "2.100", etc. which are decimal(2,1) vs decimal(4,3) datatypes.
>
> -----Original Message-----
> From: Paul Taylor <pt...@gmail.com>
> Sent: Wednesday, October 28, 2020 8:05 AM
> To: dev@arrow.apache.org
> Subject: Re: [JS] BigNum toJSON returns a string with quotations in it
>
> External Email: Use caution with links and attachments
>
>
> Yes, the quotes should be removed.
>
> I'd recommend using the binary IPC format to send data to the browser though.
>
> Deserializing to JSON is expensive, loses the benefits of features like dictionary encoding, and reconstructing types on the receiving end is error-prone (as illustrated here).
>
> Paul
>
>> On Oct 28, 2020, at 9:31 AM, Sam Peka <sa...@gmail.com> wrote:
>>
>> Hi there,
>>
>> The toJSON method of BigNum is currently returning a string with quotation marks around it: https://urldefense.com/v3/__https://github.com/apache/arrow/blob/fbb781bcdd15d54c52c71881c5c53d0c68069be6/js/src/util/bn.ts*L39__;Iw!!KSjYCgUGsB4!JQvbemEyMOKe9IdEl0I_dvM8_AKplf58CKhSvIyb0d5mHv7dJta2TJGQeGgUBlZ4tFg$
>>
>> JSON.stringify will include those quotation marks in the output JSON. Here’s an example illustrating the issue:
>>
>> const a = BN.new([12])
>> console.log(JSON.stringify({ a }))
>>> {"a":"\"12\""}
>>
>> Our usecase is to take some arrow data and return it to a web client as json – but because the format is bad we’re having to work around this by converting all BigNums to strings before serilaizing the response.
>>
>> Is BigNum.toJSON intended to work with JSON.stringify? Happy to open a PR if so.
>>
>> Best,
>> Sam
>
>
> This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to http://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy.
>
>
> For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations.
>
> © 2020 BlackRock, Inc. All rights reserved.
RE: [JS] BigNum toJSON returns a string with quotations in it
Posted by "Lee, David" <Da...@blackrock.com.INVALID>.
Quotes shouldn't be removed. BigNum is a decimal derivative and decimals in JSON have quotes.
This is needed to distinguish values like "2.1" vs "2.100", etc. which are decimal(2,1) vs decimal(4,3) datatypes.
-----Original Message-----
From: Paul Taylor <pt...@gmail.com>
Sent: Wednesday, October 28, 2020 8:05 AM
To: dev@arrow.apache.org
Subject: Re: [JS] BigNum toJSON returns a string with quotations in it
External Email: Use caution with links and attachments
Yes, the quotes should be removed.
I'd recommend using the binary IPC format to send data to the browser though.
Deserializing to JSON is expensive, loses the benefits of features like dictionary encoding, and reconstructing types on the receiving end is error-prone (as illustrated here).
Paul
> On Oct 28, 2020, at 9:31 AM, Sam Peka <sa...@gmail.com> wrote:
>
> Hi there,
>
> The toJSON method of BigNum is currently returning a string with quotation marks around it: https://urldefense.com/v3/__https://github.com/apache/arrow/blob/fbb781bcdd15d54c52c71881c5c53d0c68069be6/js/src/util/bn.ts*L39__;Iw!!KSjYCgUGsB4!JQvbemEyMOKe9IdEl0I_dvM8_AKplf58CKhSvIyb0d5mHv7dJta2TJGQeGgUBlZ4tFg$
>
> JSON.stringify will include those quotation marks in the output JSON. Here’s an example illustrating the issue:
>
> const a = BN.new([12])
> console.log(JSON.stringify({ a }))
>> {"a":"\"12\""}
>
> Our usecase is to take some arrow data and return it to a web client as json – but because the format is bad we’re having to work around this by converting all BigNums to strings before serilaizing the response.
>
> Is BigNum.toJSON intended to work with JSON.stringify? Happy to open a PR if so.
>
> Best,
> Sam
This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to http://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations.
© 2020 BlackRock, Inc. All rights reserved.
Re: [JS] BigNum toJSON returns a string with quotations in it
Posted by Paul Taylor <pt...@gmail.com>.
Yes, the quotes should be removed.
I'd recommend using the binary IPC format to send data to the browser though.
Deserializing to JSON is expensive, loses the benefits of features like dictionary encoding, and reconstructing types on the receiving end is error-prone (as illustrated here).
Paul
> On Oct 28, 2020, at 9:31 AM, Sam Peka <sa...@gmail.com> wrote:
>
> Hi there,
>
> The toJSON method of BigNum is currently returning a string with quotation marks around it: https://github.com/apache/arrow/blob/fbb781bcdd15d54c52c71881c5c53d0c68069be6/js/src/util/bn.ts#L39
>
> JSON.stringify will include those quotation marks in the output JSON. Here’s an example illustrating the issue:
>
> const a = BN.new([12])
> console.log(JSON.stringify({ a }))
>> {"a":"\"12\""}
>
> Our usecase is to take some arrow data and return it to a web client as json – but because the format is bad we’re having to work around this by converting all BigNums to strings before serilaizing the response.
>
> Is BigNum.toJSON intended to work with JSON.stringify? Happy to open a PR if so.
>
> Best,
> Sam