You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by GitBox <gi...@apache.org> on 2023/01/19 12:10:04 UTC
[GitHub] [arrow] OfekShilon opened a new issue, #33784: [R] writing/reading a data.frame with column class 'list' changes column class
OfekShilon opened a new issue, #33784:
URL: https://github.com/apache/arrow/issues/33784
### Describe the bug, including details regarding any error messages, version, and platform.
(and in addition, adds a `ptype` attribute - as already detailed in #15248)
```r
# One way to create column with class list:
library(tibble)
tb <- tibble(list_column = list(c(a = 1, b = 2)))
df <- as.data.frame(tb)
class(df$list_column)
# [1] "list"
# Write + read back
tmpf <- tempfile()
arrow::write_feather(df, tmpf)
df2 <- arrow::read_feather(tmpf)
class(df2$list_column)
# [1] "arrow_list" "vctrs_list_of" "vctrs_vctr" "list"
unlink(tmpf)
```
### Component(s)
R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] OfekShilon commented on issue #33784: [R] writing/reading a data.frame with column class 'list' changes column class
Posted by "OfekShilon (via GitHub)" <gi...@apache.org>.
OfekShilon commented on issue #33784:
URL: https://github.com/apache/arrow/issues/33784#issuecomment-1401832642
When the _column_ class changes to `c("arrow_list", "vctrs_list_of", "vctrs_vctr","list" )` rbind operations (and possibly others) break:
```r
library(tibble)
tb <- tibble(list_column = list(c(a = 1, b = 2)))
df <- as.data.frame(tb)
class(df$list_column) # [1] "list"
# Write + read back:
tmpf <- tempfile()
arrow::write_feather(df, tmpf)
df2 <- arrow::read_feather(tmpf)
class(df2$list_column) # [1] "arrow_list" "vctrs_list_of" "vctrs_vctr" "list"
rbind(df,NULL) # works
rbind(df2,NULL)
# Error in `stop_vctrs()`:
# ! `levels.arrow_list()` not supported.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] OfekShilon commented on issue #33784: [R] writing/reading a data.frame with column class 'list' changes column class
Posted by "OfekShilon (via GitHub)" <gi...@apache.org>.
OfekShilon commented on issue #33784:
URL: https://github.com/apache/arrow/issues/33784#issuecomment-1403444753
And during save of _elements_ with class "list", list names are lost.
```r
> df <- data.frame(col1=I(list(list(a=1), list(b=8))))
> df$col1[1]
[[1]]
[[1]]$a <-----
[1] 1
> tmpf <- tempfile()
> arrow::write_feather(df, tmpf)
> df2 <- arrow::read_feather(tmpf)
> df2$col1[1]
[[1]]
<list<double>[1]>
[[1]] <-----
[1] 1
> unlink(tmpf)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] OfekShilon commented on issue #33784: [R] writing/reading a data.frame with column class 'list' changes column class
Posted by "OfekShilon (via GitHub)" <gi...@apache.org>.
OfekShilon commented on issue #33784:
URL: https://github.com/apache/arrow/issues/33784#issuecomment-1441504787
@paleolimbot Thanks for the suggestions. However:
1. The `unserialize` code seems broken:
```r
...
df2 <- arrow::read_feather(tmpf)
df2$list_column <- unserialize_list_col_from_binary(df2$list_column)
# Error in FUN(X[[i]], ...) : 'connection' must be a connection
unserialize(df2$list_column)
# Error in unserialize(df2$list_column) : 'connection' must be a connection
```
2. In the `list_col_to_arrow_friendly` approach, after a save/load roundtrip I'm getting an `arrow_list` of `tibble`s, which is rather far from the list of lists I set out with.
3. In both approaches, as noted above - the class of columns returned by a save/load rountrip (`c("arrow_list","vctrs_list_of","vctrs_vctr","list")` causes `rbind` to break. This might be caused by the `vctrs` package - any insight into it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on issue #33784: [R] writing/reading a data.frame with column class 'list' changes column class
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #33784:
URL: https://github.com/apache/arrow/issues/33784#issuecomment-1415817944
Thanks for reporting this @OfekShilon. In terms of the last comment above and reported behaviour therein, some of this can be seen in #15033 too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] paleolimbot commented on issue #33784: [R] writing/reading a data.frame with column class 'list' changes column class
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #33784:
URL: https://github.com/apache/arrow/issues/33784#issuecomment-1440174266
Echoing Nic's thanks for opening this...our support for list columns is far from perfect. In particular, we drop names without warning, which we should fix.
Supporting names internally in Arrow is hard because Arrow doesn't have an internal concept of named things so we would have to invent one. We will probably get there - probably via an extension type - but in the meantime you will have to do some conversion to/from arrow yourself as a workaround.
The two workarounds I can think of off the top of my head are (1) serialize list objects on the way in and unserialize them on the way out:
``` r
library(tibble)
tb <- tibble(list_column = list(c(a = 1, b = 2)))
str(tb$list_column)
#> List of 1
#> $ : Named num [1:2] 1 2
#> ..- attr(*, "names")= chr [1:2] "a" "b"
serialize_list_col_to_binary <- function(x) {
lapply(x, serialize, NULL)
}
unserialize_list_col_from_binary <- function(x) {
lapply(x, unserialize)
}
# Write + read back:
tmpf <- tempfile()
tb$list_column <- serialize_list_col_to_binary(tb$list_column)
arrow::write_feather(tb, tmpf)
df2 <- arrow::read_feather(tmpf)
df2$list_column <- unserialize_list_col_from_binary(df2$list_column)
str(df2$list_column)
#> List of 1
#> $ : Named num [1:2] 1 2
#> ..- attr(*, "names")= chr [1:2] "a" "b"
```
...or (2) do some of your own modifications to make the list element types fit better in Arrow. In your case, your list elements could be data.frames:
``` r
library(tibble)
tb <- tibble(list_column = list(c(a = 1, b = 2)))
str(tb$list_column)
#> List of 1
#> $ : Named num [1:2] 1 2
#> ..- attr(*, "names")= chr [1:2] "a" "b"
list_col_to_arrow_friendly <- function(x) {
lapply(x, function(x) {
if (is.null(x)) NULL else as.data.frame(as.list(x))
})
}
tb$list_column <- list_col_to_arrow_friendly(tb$list_column)
str(tb$list_column)
#> List of 1
#> $ :'data.frame': 1 obs. of 2 variables:
#> ..$ a: num 1
#> ..$ b: num 2
# Write + read back:
tmpf <- tempfile()
arrow::write_feather(tb, tmpf)
df2 <- arrow::read_feather(tmpf)
str(df2$list_column)
#> list<
#> tbl_df<
#> a: double
#> b: double
#> >
#> > [1:1]
#> $ : tibble [1 × 2] (S3: tbl_df/tbl/data.frame)
#> ..$ a: num 1
#> ..$ b: num 2
#> @ ptype: tibble [0 × 2] (S3: tbl_df/tbl/data.frame)
#> ..$ a: num(0)
#> ..$ b: num(0)
```
<sup>Created on 2023-02-22 with [reprex v2.0.2](https://reprex.tidyverse.org)</sup>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] OfekShilon commented on issue #33784: [R] writing/reading a data.frame with column class 'list' changes column class
Posted by GitBox <gi...@apache.org>.
OfekShilon commented on issue #33784:
URL: https://github.com/apache/arrow/issues/33784#issuecomment-1396903597
A repro with no use of `tibble`:
```r
df <- data.frame(a=I(list(list(1), list(8))))
class(df$a[[1]])
# [1] "list"
tmpf <- tempfile()
arrow::write_feather(df, tmpf)
df2 <- arrow::read_feather(tmpf)
class(df2$a[[1]])
#[1] "arrow_list" "vctrs_list_of" "vctrs_vctr" "list"
unlink(tmpf)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org