[jira] [Created] (ARROW-2135) from_pandas improperly casting NaNs

["Matthew Gilbert (JIRA)" <ji...@apache.org>]

Matthew Gilbert created ARROW-2135:
--------------------------------------

             Summary: from_pandas improperly casting NaNs
                 Key: ARROW-2135
                 URL: https://issues.apache.org/jira/browse/ARROW-2135
             Project: Apache Arrow
          Issue Type: Bug
          Components: Python
    Affects Versions: 0.8.0
            Reporter: Matthew Gilbert


If you create a {{Table}} from a {{DataFrame}} of ints with a NaN value the NaN is improperly cast. Since pandas casts these to floats, when converted to a table the NaN is interpreted as an integer. This seems like a bug since a known limitation in pandas (the inability to have null valued integers data) is taking precedence over arrow's functionality to store these as an IntArray with nulls.

 
{code}
import pyarrow as pa
import pandas as pd

df = pd.DataFrame({"a":[1, 2, pd.np.NaN]})
schema = pa.schema([pa.field("a", pa.int64(), nullable=True)])
table = pa.Table.from_pandas(df, schema=schema)
table[0]


<pyarrow.lib.Column object at 0x7f2151d19c90>
chunk 0: <pyarrow.lib.Int64Array object at 0x7f213bf356d8>
[
  1,
  2,
  -9223372036854775808
]{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)