You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jp0317 (via GitHub)" <gi...@apache.org> on 2024/12/04 05:38:05 UTC
[PR] avoid redundant parsing of repeated value in RleDecoder [arrow-rs]
jp0317 opened a new pull request, #6834:
URL: https://github.com/apache/arrow-rs/pull/6834
# Which issue does this PR close?
No issue
# Rationale for this change
performance improvement
# What changes are included in this PR?
Moving the parsing of [repeated_value](https://github.com/apache/arrow-rs/blob/main/parquet/src/encodings/rle.rs#L379-L382) out of the loop
# Are there any user-facing changes?
Not really: although it adds `Clone` restriction to the `get_batch<T>()`, the `T` in practice always implements Clone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] avoid redundant parsing of repeated value in RleDecoder [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #6834:
URL: https://github.com/apache/arrow-rs/pull/6834#issuecomment-2516501155
Have you run any benchmarks for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] avoid redundant parsing of repeated value in RleDecoder [arrow-rs]
Posted by "jp0317 (via GitHub)" <gi...@apache.org>.
jp0317 commented on PR #6834:
URL: https://github.com/apache/arrow-rs/pull/6834#issuecomment-2518929386
> Have you run any benchmarks for this?
not really. I was doing some experiments on decoding dictionary indices and there's noticeable potion of cpu time spent on parsing this repeated value. And it seems obviously that the repeated value can be parsed outside the loop, avoiding the redundant parsing. With the change the cpu time spent inside this `get_batch` got reduced in my experiments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] avoid redundant parsing of repeated value in RleDecoder [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold merged PR #6834:
URL: https://github.com/apache/arrow-rs/pull/6834
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org