[jira] [Commented] (ARROW-1584) [PYTHON] serialize_pandas on empty dataframe

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190393#comment-16190393 ] 

ASF GitHub Bot commented on ARROW-1584:
---------------------------------------

GitHub user wesm opened a pull request:

    https://github.com/apache/arrow/pull/1155

    ARROW-1584: [C++/Python] Support Null type in IPC round trips, fix serialize_pandas on empty DataFrame

    cc @TomAugspurger @cpcloud -- this turned out to be caused by the index in the empty DataFrame having null type:
    
    ```
    In [3]: arr = pa.array([])
    
    In [4]: arr.type
    Out[4]: DataType(null)
    
    In [5]: arr = pa.array([None, None, None])
    
    In [6]: arr
    Out[6]: 
    <pyarrow.lib.NullArray object at 0x7f28d3d205e8>
    [
      NA,
      NA,
      NA
    ]
    ```
    
    This type had been unhandled in the IPC machinery, so this resolves ARROW-1637. We need to implement integration tests for this type to make sure such data can be accurately messages between Java and C++/Python

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/wesm/arrow ARROW-1584

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/arrow/pull/1155.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1155
    
----
commit ce7c7a56370eb4a957c65cfd5a26a04319658d36
Author: Wes McKinney <we...@twosigma.com>
Date:   2017-10-03T21:38:05Z

    Support Null type in IPC round trips, fix serialize_pandas on empty DataFrame
    
    Change-Id: I5dfb10ea913815d71f11743cceff731101b85d6e

----


> [PYTHON] serialize_pandas on empty dataframe
> --------------------------------------------
>
>                 Key: ARROW-1584
>                 URL: https://issues.apache.org/jira/browse/ARROW-1584
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>    Affects Versions: 0.7.0
>            Reporter: Tom Augspurger
>            Assignee: Wes McKinney
>            Priority: Minor
>              Labels: pull-request-available, python, serialization
>             Fix For: 0.8.0
>
>
> This code
> {code:python}
> import pandas as pd
> import pyarrow as pa
> pa.serialize_pandas(pd.DataFrame())
> {code}
> Raises
> {code}
> ---------------------------------------------------------------------------
> ArrowNotImplementedError                  Traceback (most recent call last)
> <ipython-input-71-ad21add45f0d> in <module>()
> ----> 1 pa.serialize_pandas(pd.DataFrame())
> ~/Envs/dask-dev/lib/python3.6/site-packages/pyarrow/ipc.py in serialize_pandas(df)
>     158     sink = pa.BufferOutputStream()
>     159     writer = pa.RecordBatchStreamWriter(sink, batch.schema)
> --> 160     writer.write_batch(batch)
>     161     writer.close()
>     162     return sink.get_result()
> pyarrow/ipc.pxi in pyarrow.lib._RecordBatchWriter.write_batch (/Users/travis/build/apache/arrow-dist/arrow/python/build/temp.macosx-10.6-intel-3.6/lib.cxx:59238)()
> pyarrow/error.pxi in pyarrow.lib.check_status (/Users/travis/build/apache/arrow-dist/arrow/python/build/temp.macosx-10.6-intel-3.6/lib.cxx:8113)()
> ArrowNotImplementedError: Unable to convert type: null
> {code}
> Presumably {{pa.deserialize_pandas}} will need a fix as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)