You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@arrow.apache.org by "Micah Kornfield (Jira)" <ji...@apache.org> on 2021/08/23 05:51:00 UTC

[jira] [Created] (ARROW-13690) [Python] Use IPC writing code for pickling RecordBatches

Micah Kornfield created ARROW-13690:
---------------------------------------

             Summary: [Python] Use IPC writing code for pickling RecordBatches
                 Key: ARROW-13690
                 URL: https://issues.apache.org/jira/browse/ARROW-13690
             Project: Apache Arrow
          Issue Type: Improvement
          Components: Python
            Reporter: Micah Kornfield


For wide schemas in particular the the recursive nature of the currently pickling algorithm for record batches makes it less efficient then using the IPC format (which can be done entirely in C++).

 

Consider switching the mechanism to use the IPC format.  I think this can be a backwards compatible change if the current leaving: _reconstruct_record_batch in place if we care about that.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)