You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Miles Granger (Jira)" <ji...@apache.org> on 2022/11/14 11:11:00 UTC

[jira] [Commented] (ARROW-18275) [Python] Allow custom reader/writer implementation for arrow dataset read/write path

    [ https://issues.apache.org/jira/browse/ARROW-18275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17633701#comment-17633701 ] 

Miles Granger commented on ARROW-18275:
---------------------------------------

Does this overlap with [ARROW-18163 | https://issues.apache.org/jira/browse/ARROW-18163]?  To me it sounds like if that were implemented, this could be accomplished by the implementation of the custom dataset format?

> [Python] Allow custom reader/writer implementation for arrow dataset read/write path
> ------------------------------------------------------------------------------------
>
>                 Key: ARROW-18275
>                 URL: https://issues.apache.org/jira/browse/ARROW-18275
>             Project: Apache Arrow
>          Issue Type: New Feature
>          Components: Python
>    Affects Versions: 10.0.0
>            Reporter: Chang She
>            Priority: Major
>              Labels: Python
>
> We're implementing a "versionable" data format where the read/write path has some metadata handling which we currently can't plug into the native pyarrow write_dataset and pa.dataset.dataset mechanism.
> What we've done currently is have our own `lance.write_dataset` and `lance.dataset` interfaces which knows about the versioning. And if you use the native arrow ones, it reads/writes an unversioned dataset.
> It would be great if:
> 1. the arrow interfaces provided a way for custom data formats to provide their own Arrow compliant reader/writer implementations, so we can delete our custom interface and stick with native pyarrow interface.
> 2. the pyarrow interface can support custom kwargs like "version=5" or "as_of=<timestamp>" or "version='latest'"
> for reference, this is what our custom C++ dataset implementation looks like: https://github.com/eto-ai/lance/blob/main/cpp/include/lance/arrow/dataset.h



--
This message was sent by Atlassian Jira
(v8.20.10#820010)