[jira] [Commented] (ARROW-1593) [PYTHON] serialize_pandas should pass through the preserve_index keyword

["Wes McKinney (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175077#comment-16175077 ] 

Wes McKinney commented on ARROW-1593:
-------------------------------------

That seems reasonable. We could also address the range index by augmenting the metadata in http://pandas-docs.github.io/pandas-docs-travis/developer.html

I have spent significantly less time optimizing {{RecordBatch.from_pandas}} than {{RecordBatch/Table.to_pandas}}, so there's likely some perf improvements that could be made on the ingest to Arrow path (e.g. parallel column conversions)

> [PYTHON] serialize_pandas should pass through the preserve_index keyword
> ------------------------------------------------------------------------
>
>                 Key: ARROW-1593
>                 URL: https://issues.apache.org/jira/browse/ARROW-1593
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Python
>    Affects Versions: 0.7.0
>            Reporter: Tom Augspurger
>            Assignee: Tom Augspurger
>            Priority: Minor
>             Fix For: 0.8.0
>
>
> I'm doing some benchmarking of Arrow serialization for dask.distributed to serialize dataframes.
> Overall things look good compared to the current implementation (using pickle). The biggest difference was pickle's ability to use pandas' RangeIndex to avoid serializing the entire Index of values when possible.
> I suspect that a "range type" isn't in scope for arrow, but in the meantime applications using Arrow could detect the `RangeIndex`, and pass {{ pyarrow.serialize_pandas(df, preserve_index=False) }} 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)