You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Phillip Cloud (JIRA)" <ji...@apache.org> on 2017/10/23 22:38:00 UTC

[jira] [Updated] (ARROW-1716) [Format/JSON] Use string integer value for Decimals in JSON

     [ https://issues.apache.org/jira/browse/ARROW-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Phillip Cloud updated ARROW-1716:
---------------------------------
    Description: 
Suprisingly, Java and C++ integration tests pass after ARROW-1588. This hides a bug, because we're writing decimal values as hex encoded bytes.

C++ and Java compare that the bytes are the same, but because C++ is interpreting everything as little endian after ARROW-1588 and Java is big endian the numbers these bytes represent will be different in their respective systems.

I propose that instead of encoding DecimaArray/DecimalVector values as hex encoded bytes, we store the integer as a string when writing Arrow DecimalArray/DecimalVector data to JSON. This will allow us to compare that the bytes have the same meaning in both systems.

This requires a change to how Arrow writes JSON.

cc [~icexelloss] [~wesmckinn] [~jnadeau]



  was:
Suprisingly, Java and C++ integration tests pass after ARROW-1588. This hides a bug, because we're writing decimal values as hex encoded bytes.

C++ and Java compare that the bytes are the same, but because C++ is interpreting everything as little endian after ARROW-1588 and Java is big endian the numbers these bytes represent will be different in their respective systems.

I propose that instead of encoding DecimaArray/DecimalVector values as hex encoded bytes, we store the integer as a string when writing Arrow data to JSON. This will allow us to compare that the bytes have the same meaning in both systems.

This requires a change to how Arrow writes JSON.

cc [~icexelloss] [~wesmckinn] [~jnadeau]




> [Format/JSON] Use string integer value for Decimals in JSON
> -----------------------------------------------------------
>
>                 Key: ARROW-1716
>                 URL: https://issues.apache.org/jira/browse/ARROW-1716
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: C++, Java - Vectors
>    Affects Versions: 0.7.1
>            Reporter: Phillip Cloud
>            Assignee: Phillip Cloud
>             Fix For: 0.8.0
>
>
> Suprisingly, Java and C++ integration tests pass after ARROW-1588. This hides a bug, because we're writing decimal values as hex encoded bytes.
> C++ and Java compare that the bytes are the same, but because C++ is interpreting everything as little endian after ARROW-1588 and Java is big endian the numbers these bytes represent will be different in their respective systems.
> I propose that instead of encoding DecimaArray/DecimalVector values as hex encoded bytes, we store the integer as a string when writing Arrow DecimalArray/DecimalVector data to JSON. This will allow us to compare that the bytes have the same meaning in both systems.
> This requires a change to how Arrow writes JSON.
> cc [~icexelloss] [~wesmckinn] [~jnadeau]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)