You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Neal Richardson (Jira)" <ji...@apache.org> on 2020/11/02 16:11:00 UTC
[jira] [Commented] (ARROW-10463) readr_to_csv_parse_options()
signature should match readr
[ https://issues.apache.org/jira/browse/ARROW-10463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224778#comment-17224778 ]
Neal Richardson commented on ARROW-10463:
-----------------------------------------
Thanks for the report. This is actually ARROW-8631--the single file reader supports this option but CSV datasets do not. The Arrow CSV reader collects options into three separate objects, and {{na}} goes into ConvertOptions: https://github.com/apache/arrow/blob/883eb572bc64430307112895976ba79df10c8c7d/r/R/csv.R#L178. But the Dataset module only knows about ReadOptions, so it tries to interpret all {{...}} args as ReadOptions https://github.com/apache/arrow/blob/883eb572bc64430307112895976ba79df10c8c7d/r/R/dataset-format.R#L125
ARROW-8631 is the ticket to implement this feature; I'll use this ticket to make the error message clearer in R in the meantime.
> readr_to_csv_parse_options() signature should match readr
> ---------------------------------------------------------
>
> Key: ARROW-10463
> URL: https://issues.apache.org/jira/browse/ARROW-10463
> Project: Apache Arrow
> Issue Type: Bug
> Components: R
> Affects Versions: 2.0.0
> Reporter: Gabriel Bassett
> Priority: Minor
>
> While read_csv_arrow()'s signature matches readr, the readr_to_csv_parse_options() function (called by way of open_dataset()) only appears to capture a subset of those options:
> (https://github.com/apache/arrow/blob/883eb572bc64430307112895976ba79df10c8c7d/r/R/csv.R#L464)
> {code:java}
> readr_to_csv_parse_options <- function(delim = ",",
> quote = '"',
> escape_double = TRUE,
> escape_backslash = FALSE,
> skip_empty_rows = TRUE){code}
> I ran into this trying to use a non-standard 'na' value:
>
> {code:java}
> open_dataset("/path/to/csv/directory/", schema = sch, partitioning=NULL, format="csv", delim=";", na="\\N", escape_backslash=TRUE, escape_double=FALSE`)
> Error in readr_to_csv_parse_options(...) : unused argument (na = "\\N")
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)