You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "rkrug (via GitHub)" <gi...@apache.org> on 2024/03/21 14:47:15 UTC

[I] Error when saving attached data.frame containing nested data.frame (character versus all NA logical column) [arrow]

rkrug opened a new issue, #40711:
URL: https://github.com/apache/arrow/issues/40711

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   When saving the attached rds file to a parquet file, I get the error as in the repress `Error: Invalid: Problem with column 3 (au_orcid): Invalid: Expecting a character vector`.
   
   The problem seems to be that one column only contains NA, and these are interpreted as logical by R, while arrow expects a character vector.
   
   ```
   arrow          15.0.1     2024-03-19 [1] https://apache.r-universe.dev (R 4.3.3)
   
   setting  value
    version  R version 4.3.3 (2024-02-29)
    os       macOS Sonoma 14.4
    system   aarch64, darwin20
    ui       X11
    language (EN)
    collate  en_US.UTF-8
    ctype    en_US.UTF-8
    tz       Europe/Zurich
    date     2024-03-21
    pandoc   3.1.12.3 @ /opt/homebrew/bin/pandoc
   ```
   
   
   ``` r
   library(arrow)
   #>
   #> Attaching package: ‘arrow’
   #> The following object is masked from ‘package:utils’:
   #>
   #>     timestamp
   data <- readRDS(“problem.rds”)
   arrow::write_dataset(
       data,
       path = “~/problem”,
       partitioning = c(“publication_year”, “page”),
       format = “parquet”
   )
   #> Error: Invalid: Problem with column 3 (au_orcid): Invalid: Expecting a character vector
   ```
   <sup>Created on 2024-03-21 with [reprex v2.1.0](https://reprex.tidyverse.org/)</sup>
   [problem.rds.zip](https://github.com/apache/arrow/files/14696782/problem.rds.zip)
   
   
   
   ### Component(s)
   
   R


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [R] Error when saving attached data.frame containing nested data.frame (character versus all NA logical column) [arrow]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #40711:
URL: https://github.com/apache/arrow/issues/40711#issuecomment-2022637845

   Thank you for reporting! A slightly more minimal reprex with a workaround. Basically, there is a nested column that is "all missing", so R probably didn't know what type to call it (and guessed logical). A workaround is to ensure that all items in a list column have an identical structure (below).
   
   ``` r
   library(arrow, warn.conflicts = FALSE)
   #> Some features are not enabled in this build of Arrow. Run `arrow_info()` for more information.
   
   df <- data.frame(
     authors = I(list(
       data.frame(name = c("Tim", "Steve"), orcid = c("123456", "7890")),
       data.frame(name = c("Rhonda", "Merv"), orcid = c(NA, NA))
     ))
   )
   
   arrow_table(df)
   #> Error: Invalid: Problem with column 2 (orcid): Invalid: Expecting a character vector
   
   # Workaround: ensure your authors column has the same schema for each item
   authors_common <- vctrs::vec_ptype_common(!!!df$authors)
   df$authors <- lapply(df$authors, vctrs::vec_cast, authors_common)
   arrow_table(df)
   #> Table
   #> 2 rows x 1 columns
   #> $authors: list<item: struct<name: string, orcid <string>>>
   #> 
   #> See $metadata for additional Schema metadata
   ```
   
   <sup>Created on 2024-03-27 with [reprex v2.1.0](https://reprex.tidyverse.org)</sup>
   
   From Arrow/R's end, this happens because our "string" converter errors when it sees a `logical()`. It probably could check to see if all items are `NA` (maybe via `anyNA()`) before erroring:
   
   https://github.com/apache/arrow/blob/a407a6b45e6121051966d699017333ce9653e958/r/src/r_to_arrow.cpp#L867-L871


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org