[jira] [Commented] (ARROW-15312) [R] filtering a dataset with is.na() misses some rows

["Nicola Crane (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-15312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482567#comment-17482567 ] 

Nicola Crane commented on ARROW-15312:
--------------------------------------

Thanks for reporting this [~Pierre Gramme] ! 

I am able to reproduce this bug on the dev version of Arrow, and it seems to be something to do with how this is being executed on datasets as I get the correct results if I then run:
{code:java}
arrow::read_parquet("test-arrow-na/part-0.parquet", as_data_frame = FALSE) %>%
  filter(is.na(y)) %>%
  collect() {code}
I'll see if anyone else has any idea what's going on here.

> [R] filtering a dataset with is.na() misses some rows
> -----------------------------------------------------
>
>                 Key: ARROW-15312
>                 URL: https://issues.apache.org/jira/browse/ARROW-15312
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: R
>    Affects Versions: 6.0.1
>         Environment: R 4.1.2 on Windows
> arrow 6.0.1
> dplyr 1.0.7
>            Reporter: Pierre Gramme
>            Priority: Major
>
> Hi !
> I just found an issue when querying an Arrow dataset with dplyr, filtering on is.na(...)
> It seems linked to columns containing only one distinct value and some NA's.
> Can you also reproduce the following?
>  
> {code:java}
>   library(arrow)
>   library(dplyr)
>   
>   ds_path = "test-arrow-na"
>   df = tibble(x=1:3, y=c(0L, 0L, NA_integer_), z=c(0L, 1L, NA_integer_))
>   
>   df %>% arrow::write_dataset(ds_path)
>   
>   # OK: Collect then filter: returns row 3, as expected
>   arrow::open_dataset(ds_path) %>% collect() %>% filter(is.na(y))
>   # ERROR: Filter then collect (on y) returns a tibble with no row
>   arrow::open_dataset(ds_path) %>% filter(is.na(y)) %>% collect()
>   
>   # OK: Filter then collect (on z) returns row 3, as expected
>   arrow::open_dataset(ds_path) %>% filter(is.na(z)) %>% collect() {code}
>  
> Thanks
> Pierre



--
This message was sent by Atlassian Jira
(v8.20.1#820001)