[jira] [Commented] (ARROW-12150) [Python] Bad type inference of mixed-precision Decimals

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-12150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17313209#comment-17313209 ] 

Joris Van den Bossche commented on ARROW-12150:
-----------------------------------------------

For the actual conversion, what behaviour would we prefer? Raise an error if a subsequent Decimal doesn't fit into the type that was inferred from the first value? 
(or first find a common scale/precision, but that would require an additional pass over the data?)

> [Python] Bad type inference of mixed-precision Decimals
> -------------------------------------------------------
>
>                 Key: ARROW-12150
>                 URL: https://issues.apache.org/jira/browse/ARROW-12150
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>    Affects Versions: 3.0.0
>         Environment: - macOS Big Sur 11.2.1
> - python 3.8.2
>            Reporter: abdel alfahham
>            Priority: Major
>             Fix For: 4.0.0
>
>
> Exporting _pyarrow.table_ that contains mixed-precision _Decimals_ using  _parquet.write_table_ creates a parquet that contains invalid data/values.
> In the example below the first value of _data_decimal_ is turned from Decimal('579.11999511718795474735088646411895751953125000000000') in the pyarrow table to Decimal('-378.68971792399258172661600550482428224218070136475136') in the parquet.
>  
> {code:java}
> import pyarrow
> from decimal import Decimal
> values_floats = [579.119995117188, 6.40999984741211, 2.0] # floats
> decs_from_values = [Decimal(v) for v in values_floats] # Decimal
> decs_from_float = [Decimal.from_float(v) for v in values_floats]
> decs_str = [Decimal(str(v)) for v in values_floats] # Decimal
> data_dict = {"data_decimal": decs_from_values, # python Decimal
>  "data_decimal_from_float": decs_from_float,
>  "data_float":values_floats, # python floats
>  "data_dec_str": decs_str}
> table = pyarrow.table(data=data_dict)
> print(table.to_pydict()) # before saving
> pyarrow.parquet.write_table(table, "./pyarrow_decimal.parquet") # saving
> print(pyarrow.parquet.read_table("./pyarrow_decimal.parquet").to_pydict()) # after saving
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)