You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "rkrug (via GitHub)" <gi...@apache.org> on 2024/03/21 14:47:15 UTC
[I] Error when saving attached data.frame containing nested data.frame (character versus all NA logical column) [arrow]
rkrug opened a new issue, #40711:
URL: https://github.com/apache/arrow/issues/40711
### Describe the bug, including details regarding any error messages, version, and platform.
When saving the attached rds file to a parquet file, I get the error as in the repress `Error: Invalid: Problem with column 3 (au_orcid): Invalid: Expecting a character vector`.
The problem seems to be that one column only contains NA, and these are interpreted as logical by R, while arrow expects a character vector.
```
arrow 15.0.1 2024-03-19 [1] https://apache.r-universe.dev (R 4.3.3)
setting value
version R version 4.3.3 (2024-02-29)
os macOS Sonoma 14.4
system aarch64, darwin20
ui X11
language (EN)
collate en_US.UTF-8
ctype en_US.UTF-8
tz Europe/Zurich
date 2024-03-21
pandoc 3.1.12.3 @ /opt/homebrew/bin/pandoc
```
``` r
library(arrow)
#>
#> Attaching package: ‘arrow’
#> The following object is masked from ‘package:utils’:
#>
#> timestamp
data <- readRDS(“problem.rds”)
arrow::write_dataset(
data,
path = “~/problem”,
partitioning = c(“publication_year”, “page”),
format = “parquet”
)
#> Error: Invalid: Problem with column 3 (au_orcid): Invalid: Expecting a character vector
```
<sup>Created on 2024-03-21 with [reprex v2.1.0](https://reprex.tidyverse.org/)</sup>
[problem.rds.zip](https://github.com/apache/arrow/files/14696782/problem.rds.zip)
### Component(s)
R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] Error when saving attached data.frame containing nested data.frame (character versus all NA logical column) [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #40711:
URL: https://github.com/apache/arrow/issues/40711#issuecomment-2022637845
Thank you for reporting! A slightly more minimal reprex with a workaround. Basically, there is a nested column that is "all missing", so R probably didn't know what type to call it (and guessed logical). A workaround is to ensure that all items in a list column have an identical structure (below).
``` r
library(arrow, warn.conflicts = FALSE)
#> Some features are not enabled in this build of Arrow. Run `arrow_info()` for more information.
df <- data.frame(
authors = I(list(
data.frame(name = c("Tim", "Steve"), orcid = c("123456", "7890")),
data.frame(name = c("Rhonda", "Merv"), orcid = c(NA, NA))
))
)
arrow_table(df)
#> Error: Invalid: Problem with column 2 (orcid): Invalid: Expecting a character vector
# Workaround: ensure your authors column has the same schema for each item
authors_common <- vctrs::vec_ptype_common(!!!df$authors)
df$authors <- lapply(df$authors, vctrs::vec_cast, authors_common)
arrow_table(df)
#> Table
#> 2 rows x 1 columns
#> $authors: list<item: struct<name: string, orcid <string>>>
#>
#> See $metadata for additional Schema metadata
```
<sup>Created on 2024-03-27 with [reprex v2.1.0](https://reprex.tidyverse.org)</sup>
From Arrow/R's end, this happens because our "string" converter errors when it sees a `logical()`. It probably could check to see if all items are `NA` (maybe via `anyNA()`) before erroring:
https://github.com/apache/arrow/blob/a407a6b45e6121051966d699017333ce9653e958/r/src/r_to_arrow.cpp#L867-L871
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org