Using ReadAt (hdfsPread) from Python

[Yevgeni Litvin <se...@gmail.com>]

I'd like to use HadoopFileSystem to open a file and then a read a block of
data specified by the offset and the length.
I see that for libhdfs driver, arrow supports hdfsPread API. It also
exposes it as ReadAt method of the RandomAccessFile interface. I did not
find don'ta way to access it from Python, though.
My question: is there a way to use ReadAt from Python and I missed it?

If not, would it make sense to extend HadoopFileSystem, and perhaps other
file systems to support that access pattern?

Thanks,

- Yevgeni

Re: Using ReadAt (hdfsPread) from Python

[Wes McKinney <we...@gmail.com>]

hi Yevgeni,

Would you like to add a `read_at` method to `pyarrow.NativeFile`? This
shouldn't be a huge patch. Can you open a JIRA issue?

Thanks

On Sat, May 18, 2019 at 3:18 AM Yevgeni Litvin <se...@gmail.com> wrote:
>
> I'd like to use HadoopFileSystem to open a file and then a read a block of
> data specified by the offset and the length.
> I see that for libhdfs driver, arrow supports hdfsPread API. It also
> exposes it as ReadAt method of the RandomAccessFile interface. I did not
> find don'ta way to access it from Python, though.
> My question: is there a way to use ReadAt from Python and I missed it?
>
> If not, would it make sense to extend HadoopFileSystem, and perhaps other
> file systems to support that access pattern?
>
> Thanks,
>
> - Yevgeni