[jira] [Created] (ARROW-10709) [Python] Difficult to make an efficient zero-copy file reader in Python

["Maarten Breddels (Jira)" <ji...@apache.org>]

Maarten Breddels created ARROW-10709:
----------------------------------------

             Summary: [Python] Difficult to make an efficient zero-copy file reader in Python
                 Key: ARROW-10709
                 URL: https://issues.apache.org/jira/browse/ARROW-10709
             Project: Apache Arrow
          Issue Type: Improvement
          Components: Python
            Reporter: Maarten Breddels


There is an option to do efficient data transport using file.read_buffer() using zero memory copies (benchmarking have confirmed that, very nice!).

However, file.read_buffer() when backed by a Python object (via PythonFile), will call PythonFile.read() via PyReadableFile::Read. A 'normal' file.read() that does memory copying, also calls the PythonFile.read() method, but only allows for a bytes object (PyBytes_Check is used in PyReadableFile::Read).
This makes it hard to create 1 file object in Python land that supports normal .read() (and thus needs to returns a bytes object) and to also support a zero-copy route where .read() can return a memory view.
Possibly the strict check on PyBytes_Check can me lifted by also allowing trying PyObject_GetBuffer.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)