You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/05/21 16:45:49 UTC
[GitHub] [arrow] lidavidm opened a new pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
lidavidm opened a new pull request #10376:
URL: https://github.com/apache/arrow/pull/10376
This exposes the same API as for Parquet fragments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-846152197
Benchmark runs are scheduled for baseline = 78c88a9f517b540ea010a6bd6a8c8f6d91769559 and contender = 8c8f05783b9282a4d5c76be971cbb5e479811499. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-large-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/9e3420617f0d41679b3fc26c9fa3b814...f3c3c436e0f141839e96f513be9abce3/)
[Scheduled] [ec2-t3-xlarge-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/e76411e0d6e14bac94541c292ce1b50a...cc398b1f047840b2af7186b28820ce6e/)
[Scheduled] [ursa-i9-9960x (mimalloc)](https://conbench.ursa.dev/compare/runs/dbca458703134cf5b2631d32f34f20a2...99eaab2223274784bca6d4ca114e84a8/)
[Scheduled] [ursa-thinkcentre-m75q (mimalloc)](https://conbench.ursa.dev/compare/runs/2e7e0fc2d0f44480ababbceee06580f0...5a4a90989e074c38b56a6faaa4d8074a/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-846152197
Benchmark runs are scheduled for baseline = 78c88a9f517b540ea010a6bd6a8c8f6d91769559 and contender = 8c8f05783b9282a4d5c76be971cbb5e479811499. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Failed :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-large-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/9e3420617f0d41679b3fc26c9fa3b814...f3c3c436e0f141839e96f513be9abce3/)
[Finished :arrow_down:0.0% :arrow_up:40.0%] [ec2-t3-xlarge-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/e76411e0d6e14bac94541c292ce1b50a...cc398b1f047840b2af7186b28820ce6e/)
[Scheduled] [ursa-i9-9960x (mimalloc)](https://conbench.ursa.dev/compare/runs/dbca458703134cf5b2631d32f34f20a2...99eaab2223274784bca6d4ca114e84a8/)
[Scheduled] [ursa-thinkcentre-m75q (mimalloc)](https://conbench.ursa.dev/compare/runs/2e7e0fc2d0f44480ababbceee06580f0...5a4a90989e074c38b56a6faaa4d8074a/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-846152197
Benchmark runs are scheduled for baseline = 78c88a9f517b540ea010a6bd6a8c8f6d91769559 and contender = 8c8f05783b9282a4d5c76be971cbb5e479811499. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-large-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/9e3420617f0d41679b3fc26c9fa3b814...f3c3c436e0f141839e96f513be9abce3/)
[Finished :arrow_down:0.0% :arrow_up:40.0%] [ec2-t3-xlarge-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/e76411e0d6e14bac94541c292ce1b50a...cc398b1f047840b2af7186b28820ce6e/)
[Scheduled] [ursa-i9-9960x (mimalloc)](https://conbench.ursa.dev/compare/runs/dbca458703134cf5b2631d32f34f20a2...99eaab2223274784bca6d4ca114e84a8/)
[Scheduled] [ursa-thinkcentre-m75q (mimalloc)](https://conbench.ursa.dev/compare/runs/2e7e0fc2d0f44480ababbceee06580f0...5a4a90989e074c38b56a6faaa4d8074a/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-875927861
@westonpace any thoughts on this? It seems to have only a small benefit at best, so maybe we should just close this instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-847979229
Benchmark runs are scheduled for baseline = 5790a80396fc39232cd66669d09c9cbd594ba6cb and contender = a817f124f58517dec53860adc2cceb9b8cf73588. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Failed :arrow_down:0.0% :arrow_up:33.33%] [ec2-t3-large-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/b89d80b242604123a4177e57bb0e27d9...5191a35934184148add49ae056ee2533/)
[Finished :arrow_down:0.0% :arrow_up:33.33%] [ec2-t3-xlarge-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/9bea11e4fa6a4d309600d5b8cbd26de6...67330c8e99244e3187c142b7c4f6fdd4/)
[Scheduled] [ursa-i9-9960x (mimalloc)](https://conbench.ursa.dev/compare/runs/fa3ec70721264c24b0d878fa43cbb578...2bd2f779c9a54a6faeaa8321766e6a72/)
[Skipped :warning: Only ['C++'] langs are supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q (mimalloc)](https://conbench.ursa.dev/compare/runs/93fa20dca6d241bc82badce82ad3f428...7619ceb782734154a3447ce51e9702be/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-846103270
https://issues.apache.org/jira/browse/ARROW-12834
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-846152098
@ursabot please benchmark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-847979229
Benchmark runs are scheduled for baseline = 5790a80396fc39232cd66669d09c9cbd594ba6cb and contender = a817f124f58517dec53860adc2cceb9b8cf73588. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Failed :arrow_down:0.0% :arrow_up:33.33%] [ec2-t3-large-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/b89d80b242604123a4177e57bb0e27d9...5191a35934184148add49ae056ee2533/)
[Finished :arrow_down:0.0% :arrow_up:33.33%] [ec2-t3-xlarge-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/9bea11e4fa6a4d309600d5b8cbd26de6...67330c8e99244e3187c142b7c4f6fdd4/)
[Finished :arrow_down:1.36% :arrow_up:0.68%] [ursa-i9-9960x (mimalloc)](https://conbench.ursa.dev/compare/runs/fa3ec70721264c24b0d878fa43cbb578...2bd2f779c9a54a6faeaa8321766e6a72/)
[Skipped :warning: Only ['C++'] langs are supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q (mimalloc)](https://conbench.ursa.dev/compare/runs/93fa20dca6d241bc82badce82ad3f428...7619ceb782734154a3447ce51e9702be/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-848045351
Conbench does show a marked difference on the i9-9960x though it seems variation is very high for that benchmark anyways when comparing to the contender.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-847979229
Benchmark runs are scheduled for baseline = 5790a80396fc39232cd66669d09c9cbd594ba6cb and contender = a817f124f58517dec53860adc2cceb9b8cf73588. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-large-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/b89d80b242604123a4177e57bb0e27d9...5191a35934184148add49ae056ee2533/)
[Finished :arrow_down:0.0% :arrow_up:33.33%] [ec2-t3-xlarge-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/9bea11e4fa6a4d309600d5b8cbd26de6...67330c8e99244e3187c142b7c4f6fdd4/)
[Scheduled] [ursa-i9-9960x (mimalloc)](https://conbench.ursa.dev/compare/runs/fa3ec70721264c24b0d878fa43cbb578...2bd2f779c9a54a6faeaa8321766e6a72/)
[Skipped :warning: Only ['C++'] langs are supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q (mimalloc)](https://conbench.ursa.dev/compare/runs/93fa20dca6d241bc82badce82ad3f428...7619ceb782734154a3447ce51e9702be/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-875932074
Unfortunately, I think I agree with closing. The "common" path for IPC is hot-in-memory reads so the actual "read" calls should be more or less a no-op. S3/IPC is also probably a pretty uncommon path. For non-S3 the kernel should be doing its own coalescing. Maybe the reason there isn't much benefit is because IPC is always doing a full read of the file so there are no real gaps. Though if we want to start optimizing for partial reads I think there is more we'd have to do than just coalesce.
So maybe a "close, but reopen and investigate further if there is a compelling use case".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-846152197
Benchmark runs are scheduled for baseline = 78c88a9f517b540ea010a6bd6a8c8f6d91769559 and contender = 8c8f05783b9282a4d5c76be971cbb5e479811499. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Failed :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-large-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/9e3420617f0d41679b3fc26c9fa3b814...f3c3c436e0f141839e96f513be9abce3/)
[Finished :arrow_down:0.0% :arrow_up:40.0%] [ec2-t3-xlarge-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/e76411e0d6e14bac94541c292ce1b50a...cc398b1f047840b2af7186b28820ce6e/)
[Scheduled] [ursa-i9-9960x (mimalloc)](https://conbench.ursa.dev/compare/runs/dbca458703134cf5b2631d32f34f20a2...99eaab2223274784bca6d4ca114e84a8/)
[Finished :arrow_down:2.15% :arrow_up:2.05%] [ursa-thinkcentre-m75q (mimalloc)](https://conbench.ursa.dev/compare/runs/2e7e0fc2d0f44480ababbceee06580f0...5a4a90989e074c38b56a6faaa4d8074a/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-847978195
Running on EC2 with more iterations, it seems there's a small benefit, with some variance (from EC2?) throwing it off.
Conbench for `pre_buffer=True`:
```
"stats": {
"batch_id": "3104564f6b8244fb814465fbbf6a19cb",
"data": [
"14.430056",
"16.669666",
"38.752780",
"15.887550",
"12.784942",
"13.485059",
"12.358206",
"12.877868",
"11.940915",
"12.250891"
],
"iqr": "3.058287",
"iterations": 10,
"max": "38.752780",
"mean": "16.143793",
"median": "13.181464",
"min": "11.940915",
"q1": "12.464890",
"q3": "15.523177",
"run_id": "3104564f6b8244fb814465fbbf6a19cb",
"stdev": "8.100836",
"time_unit": "s",
"times": [],
"timestamp": "2021-05-25T15:32:39.462551+00:00",
"unit": "s"
},
```
Conbench for `pre_buffer=False`:
```
"stats": {
"batch_id": "e09c0ea9372145ca81aab93f30c08940",
"data": [
"15.748755",
"15.441451",
"15.636802",
"15.699552",
"15.200788",
"14.816090",
"14.998590",
"14.502809",
"15.451343",
"15.484191"
],
"iqr": "0.549510",
"iterations": 10,
"max": "15.748755",
"mean": "15.298037",
"median": "15.446397",
"min": "14.502809",
"q1": "15.049140",
"q3": "15.598650",
"run_id": "e09c0ea9372145ca81aab93f30c08940",
"stdev": "0.411182",
"time_unit": "s",
"times": [],
"timestamp": "2021-05-25T15:36:19.423906+00:00",
"unit": "s"
},
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-846219969
Hmm. It's unclear if there's an improvement. Nominally there is compared to the contender but it seems the contender was just slow. I was hoping the pre_buffer=true would be faster than the pre_buffer=false within the same run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-846152596
Oh no, I forgot to scope it to just Python :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-847979109
@ursabot please benchmark lang=Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm closed pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
lidavidm closed pull request #10376:
URL: https://github.com/apache/arrow/pull/10376
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-846152197
Benchmark runs are scheduled for baseline = 78c88a9f517b540ea010a6bd6a8c8f6d91769559 and contender = 8c8f05783b9282a4d5c76be971cbb5e479811499. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Failed :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-large-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/9e3420617f0d41679b3fc26c9fa3b814...f3c3c436e0f141839e96f513be9abce3/)
[Finished :arrow_down:0.0% :arrow_up:40.0%] [ec2-t3-xlarge-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/e76411e0d6e14bac94541c292ce1b50a...cc398b1f047840b2af7186b28820ce6e/)
[Finished :arrow_down:0.71% :arrow_up:1.43%] [ursa-i9-9960x (mimalloc)](https://conbench.ursa.dev/compare/runs/dbca458703134cf5b2631d32f34f20a2...99eaab2223274784bca6d4ca114e84a8/)
[Finished :arrow_down:2.15% :arrow_up:2.05%] [ursa-thinkcentre-m75q (mimalloc)](https://conbench.ursa.dev/compare/runs/2e7e0fc2d0f44480ababbceee06580f0...5a4a90989e074c38b56a6faaa4d8074a/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-846097278
See https://github.com/ursacomputing/benchmarks/pull/45 for the accompanying benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-847979229
Benchmark runs are scheduled for baseline = 5790a80396fc39232cd66669d09c9cbd594ba6cb and contender = a817f124f58517dec53860adc2cceb9b8cf73588. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-large-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/b89d80b242604123a4177e57bb0e27d9...5191a35934184148add49ae056ee2533/)
[Scheduled] [ec2-t3-xlarge-us-east-2 (mimalloc)](https://conbench.ursa.dev/compare/runs/9bea11e4fa6a4d309600d5b8cbd26de6...67330c8e99244e3187c142b7c4f6fdd4/)
[Scheduled] [ursa-i9-9960x (mimalloc)](https://conbench.ursa.dev/compare/runs/fa3ec70721264c24b0d878fa43cbb578...2bd2f779c9a54a6faeaa8321766e6a72/)
[Skipped :warning: Only ['C++'] langs are supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q (mimalloc)](https://conbench.ursa.dev/compare/runs/93fa20dca6d241bc82badce82ad3f428...7619ceb782734154a3447ce51e9702be/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-875933569
Makes sense. I would also guess that record batches are also generally large enough that there's no need for further coalescing, unlike Parquet where the column-chunk granularity makes coalescing a highly effective optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10376: ARROW-12834: [Python] Expose pre_buffer for IPC fragments
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10376:
URL: https://github.com/apache/arrow/pull/10376#issuecomment-847102241
I'd like to test this myself on EC2 and see why we don't get any expected benefit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org