[jira] [Comment Edited] (ARROW-13887) [R] Capture error produced when reading in CSV file with headers and using a schema, and add suggestion

["Dragoș Moldovan-Grünfeld (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425014#comment-17425014 ] 

Dragoș Moldovan-Grünfeld edited comment on ARROW-13887 at 10/6/21, 2:28 PM:
----------------------------------------------------------------------------

I think there might be a bit more than meets the eye to this issue, to the point that it becomes a different conversation (about the API of the function). 
 Both `col_types` and `schema` arguments accept a `Schema` object, but the way they treat it are a bit different:
 * `schema` errors if the CSV file has column headers, while
 * `col_types` ignores it the column names. (although the documentation states that the names in the object and in the column headers *must* match, nothing happens if they don't)

A slightly modified example:


 ```
 library(arrow)

share_data <- tibble::tibble(
 company = c("AMZN", "GOOG", "BKNG", "TSLA"),
 price = c(3463.12, 2884.38, 2300.46, 732.39)
 )

tf <- tempfile()

write.csv(share_data, tf, row.names = FALSE)

share_schema <- schema(
 company = utf8(),
 price = float64()
 )
 # this errors
 read_csv_arrow(tf, schema = share_schema)

 # this works
 read_csv_arrow(tf, col_types = share_schema)

conflict_col_names_schema <- schema(
 company_test = utf8(),
 price = float64()
 )
 # this works, but as per the current documentation it should error, but
 # read_csv_arrow() simply ignores the mismatched column names (Schema object vs
 # CSV file)
 read_csv_arrow(tf, col_types = conflict_col_names_schema)

unlink(tf)
 ```


was (Author: dragosmg):
I think there might be a bit more than meets the eye to this issue, to the point that it becomes a different conversation (about the API of the function). 
Both `col_types` and `schema` arguments accept a `Schema` object, but the way they treat it are a bit different: 
* `schema` errors if the CSV file has column headers, while
* `col_types` ignores it the column names. (although the documentation states that the names in the object and in the column headers *must* match, nothing happens if they don't)

A slightly modified example:
```
library(arrow)

share_data <- tibble::tibble(
 company = c("AMZN", "GOOG", "BKNG", "TSLA"),
 price = c(3463.12, 2884.38, 2300.46, 732.39)
)

tf <- tempfile()

write.csv(share_data, tf, row.names = FALSE)

share_schema <- schema(
 company = utf8(),
 price = float64()
)

# this errors
read_csv_arrow(tf, schema = share_schema)

# this works
read_csv_arrow(tf, col_types = share_schema)

conflict_col_names_schema <- schema(
 company_test = utf8(),
 price = float64()
)

# this works, but as per the current documentation it should error, but 
# read_csv_arrow() simply ignores the mismatched column names (Schema object vs
# CSV file)
read_csv_arrow(tf, col_types = conflict_col_names_schema)

unlink(tf)
```

> [R] Capture error produced when reading in CSV file with headers and using a schema, and add suggestion
> -------------------------------------------------------------------------------------------------------
>
>                 Key: ARROW-13887
>                 URL: https://issues.apache.org/jira/browse/ARROW-13887
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: R
>            Reporter: Nicola Crane
>            Assignee: Dragoș Moldovan-Grünfeld
>            Priority: Major
>              Labels: good-first-issue
>             Fix For: 6.0.0
>
>
> When reading in a CSV with headers, and also using a schema, we get an error as the code tries to read in the header as a line of data.
> {code:java}
> share_data <- tibble::tibble(
>   company = c("AMZN", "GOOG", "BKNG", "TSLA"),
>   price = c(3463.12, 2884.38, 2300.46, 732.39)
> )
> readr::write_csv(share_data, file = "share_data.csv")
> share_schema <- schema(
>   company = utf8(),
>   price = float64()
> )
> read_csv_arrow("share_data.csv", schema = share_schema)
> {code}
> {code:java}
> Error: Invalid: In CSV column #1: CSV conversion error to double: invalid value 'price'
> /home/nic2/arrow/cpp/src/arrow/csv/converter.cc:492 decoder_.Decode(data, size, quoted, &value)
> /home/nic2/arrow/cpp/src/arrow/csv/parser.h:84 status
> /home/nic2/arrow/cpp/src/arrow/csv/converter.cc:496 parser.VisitColumn(col_index, visit) {code}
> The correct thing here would have been for the user to supply the argument {{skip=1}} to {{read_csv_arrow()}} but this is not immediately obvious from the error message returned from C++.  We should capture the error and instead supply our own error message using {{rlang::abort}} which informs the user of the error and then suggests what they can do to prevent it.
>  
> For similar examples (and their associated PRs) see {color:#1d1c1d}ARROW-11766, and ARROW-12791{color}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)