You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/10/09 18:26:00 UTC
[jira] [Commented] (ARROW-1593) [PYTHON] serialize_pandas should
pass through the preserve_index keyword
[ https://issues.apache.org/jira/browse/ARROW-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16197442#comment-16197442 ]
ASF GitHub Bot commented on ARROW-1593:
---------------------------------------
GitHub user wesm opened a pull request:
https://github.com/apache/arrow/pull/1190
ARROW-1593: [Python] Pass through preserve_index to RecordBatch.from_pandas in serialize_pandas
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wesm/arrow ARROW-1593
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/arrow/pull/1190.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1190
----
commit 04dc01713d15ee0e51739c5f56a664600aac0b62
Author: Wes McKinney <we...@twosigma.com>
Date: 2017-10-09T18:24:59Z
Pass through preserve_index to RecordBatch.from_pandas in serialize_pandas
Change-Id: I21268556e18b739db3de8601cce83388227dec0d
----
> [PYTHON] serialize_pandas should pass through the preserve_index keyword
> ------------------------------------------------------------------------
>
> Key: ARROW-1593
> URL: https://issues.apache.org/jira/browse/ARROW-1593
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Affects Versions: 0.7.0
> Reporter: Tom Augspurger
> Assignee: Wes McKinney
> Priority: Minor
> Labels: pull-request-available
> Fix For: 0.8.0
>
>
> I'm doing some benchmarking of Arrow serialization for dask.distributed to serialize dataframes.
> Overall things look good compared to the current implementation (using pickle). The biggest difference was pickle's ability to use pandas' RangeIndex to avoid serializing the entire Index of values when possible.
> I suspect that a "range type" isn't in scope for arrow, but in the meantime applications using Arrow could detect the `RangeIndex`, and pass {{ pyarrow.serialize_pandas(df, preserve_index=False) }}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)