[jira] [Comment Edited] (ARROW-11067) [R] read_csv_arrow silently fails to read some strings and returns nulls

["Weston Pace (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-11067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258777#comment-17258777 ] 

Weston Pace edited comment on ARROW-11067 at 1/5/21, 9:32 AM:
--------------------------------------------------------------

Sorry, I missed this when I was reviewing activity this morning.  I can go ahead and make a PR.  I've created a small regression for it ([https://github.com/apache/arrow/pull/9100]).  I'm not sure if it's correct.  I'm waiting to see if the Github actions can tell me.  If not I'll need to setup a quick MacOS environment (as MacOS is the only environment where int_fast16_t is 16 bits).

If you're not worried about the regression I can just push the fix and make the PR.


was (Author: westonpace):
Sorry, I missed this when I was reviewing activity this morning.  I can go ahead and make a PR.  I've created a small regression for it (https://github.com/apache/arrow/pull/9100).  I'm not sure if it's correct.  I'm waiting to see if the Github actions can tell me.  If not I'll need to setup a quick MacOS environment (as MacOS is the only environment where int_fast16_t is 16 bits).

If you're not worried about the regression I can just push the fix.

> [R] read_csv_arrow silently fails to read some strings and returns nulls
> ------------------------------------------------------------------------
>
>                 Key: ARROW-11067
>                 URL: https://issues.apache.org/jira/browse/ARROW-11067
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: R
>            Reporter: John Sheffield
>            Assignee: Antoine Pitrou
>            Priority: Major
>             Fix For: 3.0.0
>
>         Attachments: arrow_explanation.png, arrow_failure_cases.csv, arrow_failure_cases.csv, arrowbug1.png, arrowbug1.png, demo_data.csv
>
>
> A sample file is attached, showing 10 rows each of strings with consistent failures (false_na = TRUE) and consistent successes (false_na = FALSE). The strings are in the column `json_string` – if relevant, they are geojsons with min nchar of 33,229 and max nchar of 202,515.
> When I read this sample file with other R CSV readers (readr and data.table shown), the files are imported correctly and there are no NAs in the json_string column.
> When I read with arrow::read_csv_arrow, 50% of the sample json_string column end up as NAs. as_data_frame TRUE or FALSE does not change the behavior, so this might not be limited to the R interface, but I can't help debug much further upstream.
>  
>  
> {code:java}
> aaa1 <- arrow::read_csv_arrow("demo_data.csv", as_data_frame = TRUE)
> aaa2 <- arrow::read_csv_arrow("demo_data.csv", as_data_frame = FALSE)
> bbb <- data.table::fread("demo_data.csv")
> ccc <- readr::read_csv("demo_data.csv")
> mean(is.na(aaa1$json_string)) # 0.5
> mean(is.na(aaa2$column(1))) # Scalar 0.5
> mean(is.na(bbb$json_string)) # 0
> mean(is.na(ccc$json_string)) # 0{code}
>  
>  
>  * arrow 2.0 (latest CRAN)
>  * readr 1.4.0
>  * data.table 1.13.2
>  * R version 4.0.1 (2020-06-06)
>  * MacOS Catalina 10.15.7 / x86_64-apple-darwin17.0
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)