You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "RinChanNOWWW (via GitHub)" <gi...@apache.org> on 2023/09/27 02:09:22 UTC
[GitHub] [arrow-rs] RinChanNOWWW opened a new issue, #4864: improve: reuse `Arc` in parquet record batch reader.
RinChanNOWWW opened a new issue, #4864:
URL: https://github.com/apache/arrow-rs/issues/4864
Both in `arrow_reader` and `async_reader`, if there are predicates, the reader will first read arrays (wrapped in `RecordBatch`) to evaluate the predicates and get a row selection. And then the reader will use the row selection to output the final needed arrays.
If some arrays in the final output are contained in the prefetched arrays, we also have to deserialize them again. This is quite wasteful.
We should have a reasonable way to reuse the arrays.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] improve: reuse `Arc` in parquet record batch reader. [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #4864:
URL: https://github.com/apache/arrow-rs/issues/4864#issuecomment-2898107563
I am actively working on this approach. Details on
- https://github.com/apache/arrow-rs/issues/7456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] improve: reuse `Arc` in parquet record batch reader. [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #4864:
URL: https://github.com/apache/arrow-rs/issues/4864#issuecomment-2557722090
🎣 -- I believe @XiangpengHao was thinking about working on this (I know you have a paper to write too, etc lol)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #4864: improve: reuse `Arc` in parquet record batch reader.
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #4864:
URL: https://github.com/apache/arrow-rs/issues/4864#issuecomment-1737037285
> We can only keep the arrays we need to output and release the others.
That is still potentially a non-trivial amount of data, it could theoretically be an entire column chunk worth :sweat_smile:
Perhaps we could add a configurable threshold of the maximum number of bytes to keep around in this way, and fallback to decoding columns again if this threshold is exceeded?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] RinChanNOWWW commented on issue #4864: improve: reuse `Arc` in parquet record batch reader.
Posted by "RinChanNOWWW (via GitHub)" <gi...@apache.org>.
RinChanNOWWW commented on issue #4864:
URL: https://github.com/apache/arrow-rs/issues/4864#issuecomment-1737029785
> The challenge is bounding the memory usage of the reader and balancing this against the cost of decoding
We can only save the arrays we need to output and release the others.
> which for primitives is relatively low
Besides decoding there is also decompression. And it may become a large cost for variable-length type like string.
> it will probably require some sort of heuristic
Yes. Such as all the output arrays are contained in predicate arrays.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] RinChanNOWWW commented on issue #4864: improve: reuse `Arc` in parquet record batch reader.
Posted by "RinChanNOWWW (via GitHub)" <gi...@apache.org>.
RinChanNOWWW commented on issue #4864:
URL: https://github.com/apache/arrow-rs/issues/4864#issuecomment-1736561752
In my opinion, we can store the primitive arrays in the memory, and combine them into a `RecordBatch` when needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] improve: reuse `Arc` in parquet record batch reader. [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #4864:
URL: https://github.com/apache/arrow-rs/issues/4864#issuecomment-2851082110
@XiangpengHao came up with a way to cache just 2 pages, see #7363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #4864: improve: reuse `Arc` in parquet record batch reader.
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #4864:
URL: https://github.com/apache/arrow-rs/issues/4864#issuecomment-1736910868
The challenge is bounding the memory usage of the reader and balancing this against the cost of decoding, which for primitives is relatively low. I don't think it is as simple as "keep the arrays around", it will probably require some sort of heuristic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org