You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/04/13 15:30:59 UTC
[GitHub] [arrow] zeroshade opened a new pull request, #12876: ARROW-16187: [Go][Parquet] Properly utilize BufferedStream and buffer size when reading
zeroshade opened a new pull request, #12876:
URL: https://github.com/apache/arrow/pull/12876
Currently the BufferSize in the `ReaderProperties` isn't utilized properly when enabling BufferedStreams. This fixes that issue so that enabling `BufferedStream` reading via the properties will correctly utilize the given buffer size when reading. The default buffersize is currently 16K, so reads that are larger than that will ignore the buffering and just pull directly from the underlying reader when BufferedStream is enabled, pulling the entire page or otherwise from the reader if BufferedStream is not enabled.
The buffer size can be set larger so that controlled reads can improve performance on high-latency readers without having to use the memory to read the entire column/page/row group into memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #12876: ARROW-16187: [Go][Parquet] Properly utilize BufferedStream and buffer size when reading
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #12876:
URL: https://github.com/apache/arrow/pull/12876#issuecomment-1098258667
https://issues.apache.org/jira/browse/ARROW-16187
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #12876: ARROW-16187: [Go][Parquet] Properly utilize BufferedStream and buffer size when reading
Posted by GitBox <gi...@apache.org>.
emkornfield commented on PR #12876:
URL: https://github.com/apache/arrow/pull/12876#issuecomment-1103378245
+1 LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade closed pull request #12876: ARROW-16187: [Go][Parquet] Properly utilize BufferedStream and buffer size when reading
Posted by GitBox <gi...@apache.org>.
zeroshade closed pull request #12876: ARROW-16187: [Go][Parquet] Properly utilize BufferedStream and buffer size when reading
URL: https://github.com/apache/arrow/pull/12876
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #12876: ARROW-16187: [Go][Parquet] Properly utilize BufferedStream and buffer size when reading
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #12876:
URL: https://github.com/apache/arrow/pull/12876#issuecomment-1107966614
Benchmark runs are scheduled for baseline = cbf7d76a5945dc665a3272507893685036a7dcb0 and contender = 8bd5514f52bf9cc542a389edaf697cbc2c97b752. 8bd5514f52bf9cc542a389edaf697cbc2c97b752 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/1912c3fc245c47c481a8cf8e78447ec9...3da722dffc1c425fbb1baacf246902af/)
[Failed] [test-mac-arm](https://conbench.ursa.dev/compare/runs/8092c3f4e13e494bb0c0fd6499460ca4...6f7e0408739e4939b45cf0c079f6cbc7/)
[Failed :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/2e31dc349f8f40e3bb228655d98dbdda...6eb4313b43a24b69b8f6e95ac540e12e/)
[Finished :arrow_down:0.34% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/24eb67022e954f30a0fc34c061b8313b...b33fa74d1d544a3b8dd5724f670dd4c4/)
Buildkite builds:
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/580| `8bd5514f` ec2-t3-xlarge-us-east-2>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/568| `8bd5514f` test-mac-arm>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/566| `8bd5514f` ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/578| `8bd5514f` ursa-thinkcentre-m75q>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/579| `cbf7d76a` ec2-t3-xlarge-us-east-2>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/567| `cbf7d76a` test-mac-arm>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/565| `cbf7d76a` ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/577| `cbf7d76a` ursa-thinkcentre-m75q>
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org