You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/10/26 15:06:53 UTC
[GitHub] [arrow] paleolimbot opened a new pull request, #14521: ARROW-18012: [R] Make map_batches .lazy = TRUE by default
paleolimbot opened a new pull request, #14521:
URL: https://github.com/apache/arrow/pull/14521
This makes the default `map_batches()` behaviour lazy (i.e., the function is called once per batch as each batch arrives):
``` r
library(arrow, warn.conflicts = FALSE)
#> Some features are not enabled in this build of Arrow. Run `arrow_info()` for more information.
source <- RecordBatchReader$create(
record_batch(a = 1:10),
record_batch(a = 11:20)
)
mapped <- map_batches(source, function(x) {
message("Hi! I'm being evaluated!")
x
}, .schema = source$schema)
as_arrow_table(mapped)
#> Hi! I'm being evaluated!
#> Hi! I'm being evaluated!
#> Table
#> 20 rows x 1 columns
#> $a <int32>
```
<sup>Created on 2022-10-26 with [reprex v2.0.2](https://reprex.tidyverse.org)</sup>
This was previously a confusing default since piping the resulting `RecordBatchReader` into an `ExecPlan` would fail for some ExecPlans before ARROW-17178 (#13706). This PR commits to the (more optimal/expected) lazy behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #14521: ARROW-18012: [R] Make map_batches .lazy = TRUE by default
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #14521:
URL: https://github.com/apache/arrow/pull/14521#issuecomment-1296294993
Benchmark runs are scheduled for baseline = 286c263492860bf6d62b3e39c80147b787848020 and contender = 97076308d07e447ad52fd4fa026f8d92513b98c9. 97076308d07e447ad52fd4fa026f8d92513b98c9 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/5bdbc69199164a4a8632a3e7c9313346...4cfffcc448424d0d91d614f6dfc8e67e/)
[Failed :arrow_down:0.0% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/e2ea5bf67dc64cec93a6082592651489...c5fb6d653f254a37945a8c84d2bf8f90/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/06409c55820d43b492cca807bcbc15c5...89610ef53ca44b9a920b5a7579751b72/)
[Finished :arrow_down:0.21% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/d0ea4ebfe19444698965ff7348e2536a...cfcb08037e2943a088579607d3158690/)
Buildkite builds:
[Finished] [`97076308` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1789)
[Failed] [`97076308` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1810)
[Finished] [`97076308` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1777)
[Finished] [`97076308` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1802)
[Finished] [`286c2634` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1788)
[Failed] [`286c2634` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1809)
[Finished] [`286c2634` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1776)
[Finished] [`286c2634` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1801)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #14521: ARROW-18012: [R] Make map_batches .lazy = TRUE by default
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14521:
URL: https://github.com/apache/arrow/pull/14521#issuecomment-1292399120
https://issues.apache.org/jira/browse/ARROW-18012
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #14521: ARROW-18012: [R] Make map_batches .lazy = TRUE by default
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14521:
URL: https://github.com/apache/arrow/pull/14521#issuecomment-1292399157
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] paleolimbot merged pull request #14521: ARROW-18012: [R] Make map_batches .lazy = TRUE by default
Posted by GitBox <gi...@apache.org>.
paleolimbot merged PR #14521:
URL: https://github.com/apache/arrow/pull/14521
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org