[jira] [Created] (ARROW-18163) [python] registering new data formats

["Chang She (Jira)" <ji...@apache.org>]

Chang She created ARROW-18163:
---------------------------------

             Summary: [python] registering new data formats
                 Key: ARROW-18163
                 URL: https://issues.apache.org/jira/browse/ARROW-18163
             Project: Apache Arrow
          Issue Type: New Feature
          Components: Python
    Affects Versions: 9.0.0
            Reporter: Chang She


Context: we're creating a new data format for computer vision (https://github.com/eto-ai/lance) with a C++ core.

We've implemented the integration so you can read Lance datasets into pyarrow like:

```python
import lance
import pyarrow.dataset as ds

ds.dataset(uri, format=lance.LanceFileFormat())
```

Would it possible to create a file format registry? like: 

```python
ds.register_file_format(
    ext='lance', 
    format=lance.LanceFileFormat(),
    dataset=lance.FileSystemDataset
)
```

which would enable: `ds.dataset('/my/dataset.lance')` to execute successfully.


The optional third argument would be to help expose format specific optimizations. e.g, Lance has much better random access performance so pushing limit/offset parameters to Lance allows for much faster paging, especially over deeply nested data and/or image blobs.

Thanks!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)