[jira] [Created] (ARROW-2205) Option for integer object nulls

["Albert Shieh (JIRA)" <ji...@apache.org>]

Albert Shieh created ARROW-2205:
-----------------------------------

             Summary: Option for integer object nulls
                 Key: ARROW-2205
                 URL: https://issues.apache.org/jira/browse/ARROW-2205
             Project: Apache Arrow
          Issue Type: New Feature
          Components: C++, Python
            Reporter: Albert Shieh


I have a use case where the loss of precision in casting integers to floats matters, and pandas supports storing integers with nulls without loss of precision in object columns. However, a roundtrip through arrow will cast the object columns to float columns, even though the object columns are stored in arrow as integers with nulls.

This is a minimal example demonstrating the behavior of a roundtrip:
{code}
import numpy as np
import pandas as pd
import pyarrow as pa

df = pd.DataFrame({"a": np.array([None, 1], dtype=object)})
df_pa = pa.Table.from_pandas(df).to_pandas()

print(df)
print(df_pa)
{code}
The output is:
{code}
      a
0  None
1     1
     a
0  NaN
1  1.0
{code}
This seems to be the desired behavior, given test_int_object_nulls in test_convert_pandas.

I think it would be useful to add an option in the to_pandas methods to allow integers with nulls to be returned as object columns. The option can default to false in order to preserve the current behavior.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)