You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "stevbear (via GitHub)" <gi...@apache.org> on 2025/02/12 22:29:37 UTC
[I] How to best optimize reading from S3? [arrow-go]
stevbear opened a new issue, #278:
URL: https://github.com/apache/arrow-go/issues/278
### Describe the usage question you have. Please include as many useful details as possible.
Hi!
I have a use case of reading certain row groups from S3.
I see that there is an option BufferedStreamEnabled.
When I set BufferedStreamEnabled to false, it seems to try to read all of the data of a column for a row group at once, which will, unfortunately, result in OOM for us.
When I set BufferedStreamEnabled to true, the library seems to be reading the row group page by page, which is not optimal for cloud usage.
How can I improve this? I imagine that the best way to improve this would be to read multiple pages in one read() sys call?
### Component(s)
Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to best optimize reading from S3? [arrow-go]
Posted by "stevbear (via GitHub)" <gi...@apache.org>.
stevbear commented on issue #278:
URL: https://github.com/apache/arrow-go/issues/278#issuecomment-2670745686
Thank you @zeroshade, this looks good to me!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to best optimize reading from S3? [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #278:
URL: https://github.com/apache/arrow-go/issues/278#issuecomment-2741136394
When using `SeekToRow` one of two things will occur:
1. If the file has an offset index populated, then it will use the offset index to find the right location to start reading from. so it will *only* load the relevant records.
2. If the file doesn't have an offset index then it will still need to read the page headers to find the right page to start reading from. So it'll avoid reading the body of the pages that are skipped.
That all said, this is specifically if you have BufferedStreamEnabled set in the reader properties (so that it uses buffered IO). If you do not enable the buffered stream, it will always read the entire column into memory before it starts skipping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to best optimize reading from S3? [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade closed issue #278: How to best optimize reading from S3?
URL: https://github.com/apache/arrow-go/issues/278
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to best optimize reading from S3? [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #278:
URL: https://github.com/apache/arrow-go/issues/278#issuecomment-2657404621
Hi @stevbear
Optimizing the reading from S3 has been on my list for a while and I just hadn't gotten around to it. It didn't get prioritized because no one had filed any issues concerning it, so thank you for filing this!
> When I set BufferedStreamEnabled to true, the library seems to be reading the row group page by page, which is not optimal for cloud usage.
This is likely because the default buffer size is 16KB, have you tried increasing the `BufferSize` member of the `ReaderProperties` so that it buffers more pages at a time? Perhaps try a buffer of a few MB?
I have a few ideas to further optimize via pre-buffering (which all have different trade-offs) so can you give me a bit more context to make sure that your use case would be helped and to identify which idea would work best for you?
Specifically: If reading an entire column for a single row group gives you OOM, you either have a significantly large row group, or I'm guessing it's string data with a lot of large strings? That leads to the question of what you're doing with the column data after you read it from the row group. If you can't hold the entire column in memory from a single row group, are you streaming the data somewhere? Are you reading only a single column at a time or multiple columns from the row group? Can you give me more of an idea of the sizes of the columns / row group of the file and the memory limitations of your system?
Is the issue the copy that happens when decoding/decompressing the column data? etc.
The more information the better so we can figure out a good solution here, gives me the opportunity to improve the memory usage of the parquet package like I've been wanting to! :smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to best optimize reading from S3? [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #278:
URL: https://github.com/apache/arrow-go/issues/278#issuecomment-2737058701
Is there any particular need for that with a `ReadTable` function as opposed to adding SeekToRow on the `RowGroupReader`? or otherwise enhancing the RecordReader?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to best optimize reading from S3? [arrow-go]
Posted by "Sunyue (via GitHub)" <gi...@apache.org>.
Sunyue commented on issue #278:
URL: https://github.com/apache/arrow-go/issues/278#issuecomment-2735518194
> ng with repeated columns a `SeekToRow` doesn't make so much sense with the low level APIs (distinguishing rows vs values is not easy for skipping within a page).
> I started implementing this a bit and realized that when you start dealing with repeated columns a `SeekToRow` doesn't make so much sense with the low level APIs (distinguishing rows vs values is not easy for skipping within a page).
>
> It makes more sense with the Arrow column and record readers in the `pqarrow` packages. What do you think?
Hi @zeroshade I have similar use case and I came cross this issue. Like you mentioned here, I find it makes more sense to enhance `pqarrow` packages, e.g. enhance `RowGroupReader.ReadTable` to something like `func (rgr RowGroupReader) ReadTable(ctx context.Context, colIndices []int, offset, numOfRecords int) (arrow.Table, error)` This way we can read partial of the row group into memory, which makes it more memory efficient. Do you think it would be possible? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to best optimize reading from S3? [arrow-go]
Posted by "stevbear (via GitHub)" <gi...@apache.org>.
stevbear commented on issue #278:
URL: https://github.com/apache/arrow-go/issues/278#issuecomment-2657578697
I see. Thanks for that explanation.
Here's an example: given a row group, containing pages:
- page 0 (min: 1, max: 5)
- page 1 (min: 6, max: 9)
- page 2 (min: 10, max: 16)
- page 3 (min: 17, max: 22)
We would like to read a range of data from 7 to 14.
We should be able to use column indexes and page indexes to narrow the pages down to just page 1 and page 2, and read those pages.
I think parquet-go (https://github.com/parquet-go/parquet-go) implemented an API to seek to a row number within the row group and an API to read x number of rows from there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to best optimize reading from S3? [arrow-go]
Posted by "stevbear (via GitHub)" <gi...@apache.org>.
stevbear commented on issue #278:
URL: https://github.com/apache/arrow-go/issues/278#issuecomment-2657548856
Thanks! That makes complete sense!
Do you have an example handy of using column indexes and offset indexes and reading pages?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to best optimize reading from S3? [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #278:
URL: https://github.com/apache/arrow-go/issues/278#issuecomment-2657493933
The BufferSize matters because it controls the underlying `BufferedReader` that is being read from, (see https://github.com/apache/arrow-go/blob/main/internal/utils/buf_reader.go#L72). When the page reader attempts to read from S3 for the page header and data, the underlying `BufferedReader` will first read `BufferSize` data from S3 to fill the buffer and the buffers that are created/allocated are merely copying from that internal buffer.
> In the current implementation, what I have observed is that, for those pages, it will require 1 read for the page header, and another read for the content (sometimes this doesn't happen if reading the page header covers the content). I was thinking that, since S3 has high latency, it would be more efficient to support a mechanism of reading 2 or more pages at the same time, given that memory will fit.
Assuming the `BufferSize` is large enough and `BufferedStreamEnabled` is true, it should read multiple pages at once for you. Another option would be to enable providing a read-cache that you could pre-populate by using the OffsetIndex and PageLocations if you know which pages you want. That said, right now the current APIs don't make it easy (or really possible) to have the page reader skip entire pages or read specific pages and take advantage of such a read-cache. So In addition to the read-cache I'd have to add changes to make that more possible.
Does that make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to best optimize reading from S3? [arrow-go]
Posted by "stevbear (via GitHub)" <gi...@apache.org>.
stevbear commented on issue #278:
URL: https://github.com/apache/arrow-go/issues/278#issuecomment-2657436043
Hi @zeroshade, thanks for getting back to me.
The OOM is not my concern at the moment, as I was able to get away with using BufferSize and specifying a reasonable batch size. My concern is more on efficiency. The BufferSize inside of ReaderProperties doesn't matter that much If I'm reading the source code correctly, as it creates a new buffer when reading the page data (https://github.com/apache/arrow-go/blob/main/parquet/file/page_reader.go#L543).
We have a not so usual use case of using parquet files for point reads, meaning that, on each read, we will only be reading a few hundred rows, that is in one, or 2 - 3 consecutive pages. Looking at the implementation, I will be able to use column indexes and page indexes to narrow down the pages I will need. In the current implementation, what I have observed is that, for those pages, it will require 1 read for the page header, and another read for the content (sometimes this doesn't happen if reading the page header covers the content). I was thinking that, since S3 has high latency, it would be more efficient to support a mechanism of reading 2 or more pages at the same time, given that memory will fit.
Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to best optimize reading from S3? [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #278:
URL: https://github.com/apache/arrow-go/issues/278#issuecomment-2660588768
I started implementing this a bit and realized that when you start dealing with repeated columns a `SeekToRow` doesn't make so much sense with the low level APIs (distinguishing rows vs values is not easy for skipping within a page).
It makes more sense with the Arrow column and record readers in the `pqarrow` packages. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to best optimize reading from S3? [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #278:
URL: https://github.com/apache/arrow-go/issues/278#issuecomment-2666429815
@stevbear can you take a look at the linked PR and let me know if this works for you or if we need more work on this front?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to best optimize reading from S3? [arrow-go]
Posted by "Sunyue (via GitHub)" <gi...@apache.org>.
Sunyue commented on issue #278:
URL: https://github.com/apache/arrow-go/issues/278#issuecomment-2738945466
> Is there any particular need for that with a `ReadTable` function as opposed to adding SeekToRow on the `RowGroupReader`? or otherwise enhancing the RecordReader?
Both sounds really good :) Our goal is to read a arrow table at given offset for X rows from the parquet file.
By the way, if it can be implemented, will the whole row group being loaded into the memory in the first place or it is possible to only load the relevant records?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to best optimize reading from S3? [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #278:
URL: https://github.com/apache/arrow-go/issues/278#issuecomment-2657556207
That's where the current problem is, as it currently stands there isn't a good way to actually *use* the column and offset indexes when reading pages as right now there isn't a way to tell the page reader to skip an entire page or to utilize the pagelocation/offset information to go to a specific page.
It would be helpful if you had an example of your use case scenario, and we could potentially work together to figure out what a good new API would look like to add support for leveraging the indexes to skip pages etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org