You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@arrow.apache.org by "Krisztian Szucs (Jira)" <ji...@apache.org> on 2020/07/02 18:01:00 UTC

[jira] [Commented] (ARROW-8651) [Python][Dataset] Support pickling of Dataset objects

    [ https://issues.apache.org/jira/browse/ARROW-8651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150483#comment-17150483 ] 

Krisztian Szucs commented on ARROW-8651:
----------------------------------------

Since FileFragment supports file handlers/buffers it makes the possible implementation a bit more complicated. Certain components of the dataset API are already serializable on the C++ side, so I assume we should make that implementation complete before adding pickling for DataFragments.  cc [~bkietz]

> [Python][Dataset] Support pickling of Dataset objects
> -----------------------------------------------------
>
>                 Key: ARROW-8651
>                 URL: https://issues.apache.org/jira/browse/ARROW-8651
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Python
>            Reporter: Joris Van den Bossche
>            Assignee: Krisztian Szucs
>            Priority: Major
>              Labels: dataset
>             Fix For: 1.0.0
>
>
> We alraedy made several parts of a Dataset serializable (the formats, the expressions, the filesystem). With those, it should also be possible to pickle FileFragments, and with that also Dataset.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)