[GitHub] [arrow] wjones127 commented on issue #12919: [R Package] map_batches in parallel

[GitBox <gi...@apache.org>]

wjones127 commented on issue #12919:
URL: https://github.com/apache/arrow/issues/12919#issuecomment-1133047634

   > Am I right map_batches right now supports single core processing as it is implemented as a while loop over the batches?
   
   Yes, right now it only uses a single core. Other operations implemented in C++ (like the dplyr ones) are multi-threaded, but I don't think we have a way to arbitrary R code in multiple threads.
   
   > Do you plan to bring this very handy capability to R arrow, too?
   
   Maybe. From [Futures](https://cran.r-project.org/web/packages/future/index.html) package:
   
    > This package implements sequential, multicore, multisession, and cluster futures. With these, R expressions can be evaluated on the local machine, in parallel a set of local machines, or distributed on a mix of local and remote machines.
   
   I'm not sure if we would target non-local execution, but we might target multisession if we had a good way of sharing the Arrow RecordBatches with subprocesses easily.
   
   I might research this a bit. @paleolimbot you have any thoughts on this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org