You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "etiennebacher (via GitHub)" <gi...@apache.org> on 2024/02/26 11:56:27 UTC

[I] `as_arrow_table()` fails when called in `future_lapply()` with `plan(multisession)` [arrow]

etiennebacher opened a new issue, #40231:
URL: https://github.com/apache/arrow/issues/40231

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   `arrow` produces invalid `arrow_table` when it is called in a `multisession` plan created by [`future`](https://www.futureverse.org/) (or its related packages). In the examples below, `future_lapply()` fails when the plan is `multisession` but not when the plan is `sequential`. I added a simple `lapply()` call to show that it works normally and that the problem really comes from `future_lapply()`.
   
   * `plan(sequential)`
   
   ``` r
   library(arrow, warn.conflicts = FALSE)
   library(future.apply)
   #> Loading required package: future
   
   plan(sequential)
   
   lapply(1, \(x) as_arrow_table(iris))
   #> [[1]]
   #> Table
   #> 150 rows x 5 columns
   #> $Sepal.Length <double>
   #> $Sepal.Width <double>
   #> $Petal.Length <double>
   #> $Petal.Width <double>
   #> $Species <dictionary<values=string, indices=int8>>
   #> 
   #> See $metadata for additional Schema metadata
   future_lapply(1, \(x) as_arrow_table(iris))
   #> [[1]]
   #> Table
   #> 150 rows x 5 columns
   #> $Sepal.Length <double>
   #> $Sepal.Width <double>
   #> $Petal.Length <double>
   #> $Petal.Width <double>
   #> $Species <dictionary<values=string, indices=int8>>
   #> 
   #> See $metadata for additional Schema metadata
   ```
   
   * `plan(multisession)`
   
   ``` r
   library(arrow, warn.conflicts = FALSE)
   library(future.apply)
   #> Loading required package: future
   
   plan(multisession)
   
   lapply(1, \(x) as_arrow_table(iris))
   #> [[1]]
   #> Table
   #> 150 rows x 5 columns
   #> $Sepal.Length <double>
   #> $Sepal.Width <double>
   #> $Petal.Length <double>
   #> $Petal.Width <double>
   #> $Species <dictionary<values=string, indices=int8>>
   #> 
   #> See $metadata for additional Schema metadata
   future_lapply(1, \(x) as_arrow_table(iris))
   #> [[1]]
   #> Table
   #> Error: Invalid <Table>, external pointer to null
   ```
   
   ---
   
   This failure with `future_lapply()` was originally found in `polars`: https://github.com/pola-rs/r-polars/issues/851
   
   cc @eitsupi 
   
   
   
   
   ### Component(s)
   
   R


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [R] `as_arrow_table()` fails when called in `future_lapply()` with `plan(multisession)` [arrow]

Posted by "shikokuchuo (via GitHub)" <gi...@apache.org>.
shikokuchuo commented on issue #40231:
URL: https://github.com/apache/arrow/issues/40231#issuecomment-1967047575

   Hi @etiennebacher @eitsupi 
   
   It is true that objects backed by external pointers cannot be straightforwardly exported to parallel processes.
   
   However [{mirai}](https://github.com/shikokuchuo/mirai) has devised a novel method, utilising the low level 'refhook' capability of R serialization itself, to allow such objects to be used transparently in the same way as other R objects.
   
   This was originally devised for torch tensors: https://shikokuchuo.net/mirai/articles/torch.html
   
   This has just been extended in the development version to support other object types such as Arrow.
   
   ``` r
   library(arrow, warn.conflicts = FALSE)
   library(mirai)
   serialization(refhook = list(arrow::write_to_raw, arrow::read_ipc_stream), class = "ArrowTabular")
   cl <- make_cluster(1)
   parallel::parLapply(cl, 1, \(x) arrow::as_arrow_table(head(iris)))
   #> [[1]]
   #>   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
   #> 1          5.1         3.5          1.4         0.2  setosa
   #> 2          4.9         3.0          1.4         0.2  setosa
   #> 3          4.7         3.2          1.3         0.2  setosa
   #> 4          4.6         3.1          1.5         0.2  setosa
   #> 5          5.0         3.6          1.4         0.2  setosa
   #> 6          5.4         3.9          1.7         0.4  setosa
   ```
   
   The above aims to demonstrate something close to what you were attempting above.
   
   It relies on registering the custom serialization and unserialisation functions. Here the return value is a 'tibble' as this is what is produced round-tripping `read_ipc_stream(write_ipc_raw(x))` but as per the torch case, it is possible to get back the same type if this were perfect.
   
   Using the native 'mirai' interface better demonstrates the possibilities, such as seamlessly moving Arrow objects in deeply nested structures:
   
   ``` r
   library(arrow, warn.conflicts = FALSE)
   library(mirai)
   serialization(refhook = list(arrow::write_to_raw, arrow::read_ipc_stream), class = "ArrowTabular")
   daemons(2)
   #> [1] 2
   m <- mirai(list(a = arrow::as_arrow_table(x), b = "some text"), x = arrow::as_arrow_table(head(iris)))
   call_mirai(m)$data
   #> $a
   #>   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
   #> 1          5.1         3.5          1.4         0.2  setosa
   #> 2          4.9         3.0          1.4         0.2  setosa
   #> 3          4.7         3.2          1.3         0.2  setosa
   #> 4          4.6         3.1          1.5         0.2  setosa
   #> 5          5.0         3.6          1.4         0.2  setosa
   #> 6          5.4         3.9          1.7         0.4  setosa
   #> 
   #> $b
   #> [1] "some text"
   ```
   Hope you find this useful. If you believe there is a better way to integrate with Arrow, please do let me know.
   
   p.s. you'll need the dev versions installable from:
   ```r
   install.packages(c("nanonext", "mirai"), repos = "hibiki-ai.r-universe.dev")
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [R] `as_arrow_table()` fails when called in `future_lapply()` with `plan(multisession)` [arrow]

Posted by "etiennebacher (via GitHub)" <gi...@apache.org>.
etiennebacher commented on issue #40231:
URL: https://github.com/apache/arrow/issues/40231#issuecomment-1966024098

   Actually, this looks like a documented limitation of `future` about exporting pointers: https://future.futureverse.org/articles/future-4-non-exportable-objects.html?q=pointer#packages-that-rely-on-external-pointers
   
   Setting `options(future.globals.onReference = "error")` shows the error:
   
   ``` r
   library(arrow, warn.conflicts = FALSE)
   library(future.apply)
   #> Loading required package: future
   
   plan(multisession)
   
   options(future.globals.onReference = "error")  
   future_lapply(1, \(x) as_arrow_table(iris))
   #> Error: Detected a non-exportable reference ('externalptr') in the value (of class 'list') of the resolved future
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [R] `as_arrow_table()` fails when called in `future_lapply()` with `plan(multisession)` [arrow]

Posted by "etiennebacher (via GitHub)" <gi...@apache.org>.
etiennebacher closed issue #40231: [R] `as_arrow_table()` fails when called in `future_lapply()` with `plan(multisession)`
URL: https://github.com/apache/arrow/issues/40231


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [R] `as_arrow_table()` fails when called in `future_lapply()` with `plan(multisession)` [arrow]

Posted by "HenrikBengtsson (via GitHub)" <gi...@apache.org>.
HenrikBengtsson commented on issue #40231:
URL: https://github.com/apache/arrow/issues/40231#issuecomment-1970341990

   FYI, you can apply those marshalling and unmarshalling operations manually, e.g.
   
   ```r
   library(parallel)
   cl <- makeCluster(1)
   
   .y <- parallel::parLapply(cl, X = 1, fun = function(x) {
     ## Create arrow table on parallel worker
     z <- arrow::as_arrow_table(head(iris))
   
     ## Marshal non-exportable objects
     arrow::write_to_raw(z)
   })
   
   ## Unmarshal marshalled objects
   y <- lapply(.y, FUN = arrow::read_ipc_stream)
   
   print(y)
   #> [[1]]
   #> # A tibble: 6 × 5
   #>   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
   #>          <dbl>       <dbl>        <dbl>       <dbl> <fct>  
   #> 1          5.1         3.5          1.4         0.2 setosa 
   #> 2          4.9         3            1.4         0.2 setosa 
   #> 3          4.7         3.2          1.3         0.2 setosa 
   #> 4          4.6         3.1          1.5         0.2 setosa 
   #> 5          5           3.6          1.4         0.2 setosa 
   #> 6          5.4         3.9          1.7         0.4 setosa 
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [R] `as_arrow_table()` fails when called in `future_lapply()` with `plan(multisession)` [arrow]

Posted by "etiennebacher (via GitHub)" <gi...@apache.org>.
etiennebacher commented on issue #40231:
URL: https://github.com/apache/arrow/issues/40231#issuecomment-1967387909

   Thanks for the info @shikokuchuo. I'm unlikely to try this soon since I don't use `future` or `mirai` (the issue above was triggered by a user report) but that's good to know


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org