[jira] [Comment Edited] (ARROW-13169) [R] group_by + write_dataset skips some countries with UN COMTRADE / BACI datasets

["Mauricio 'Pachá' Vargas Sepúlveda (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17369036#comment-17369036 ] 

Mauricio 'Pachá' Vargas Sepúlveda edited comment on ARROW-13169 at 6/24/21, 6:46 PM:
-------------------------------------------------------------------------------------

thanks, Jon.
Just to confirm that 4.0.1 from CRAN works ok, the most recent night doesn't work weel at this task

Here's USA (And many other countries) with 4.0.1
 !screenshot-1.png! 

I installed with
{code}
~$ export LIBARROW_MINIMAL=false
~$ export LIBARROW_BINARY=true
source("https://raw.githubusercontent.com/apache/arrow/master/r/R/install-arrow.R")
install_arrow()
{code}


was (Author: pachamaltese):
thanks, Jon.
Just to confirm that 4.0.1 from CRAN works ok:
 !screenshot-1.png! 

I installed with
{code}
~$ export LIBARROW_MINIMAL=false
~$ export LIBARROW_BINARY=true
source("https://raw.githubusercontent.com/apache/arrow/master/r/R/install-arrow.R")
install_arrow()
{code}

> [R] group_by + write_dataset skips some countries with UN COMTRADE / BACI datasets
> ----------------------------------------------------------------------------------
>
>                 Key: ARROW-13169
>                 URL: https://issues.apache.org/jira/browse/ARROW-13169
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: R
>            Reporter: Mauricio 'Pachá' Vargas Sepúlveda
>            Priority: Major
>             Fix For: 5.0.0
>
>         Attachments: screenshot-1.png
>
>
> A bit of context: the data for this  example contains all the world exports in 1995, it contain 212 countries, but when saving it as parquet, only 66 countries are actually recorded. The verification I included was to check if the USA (one of the best in the reporter quality index) was present in the data.
> {code:r}
> library(arrow)
> #> 
> #> Attaching package: 'arrow'
> #> The following object is masked from 'package:utils':
> #> 
> #>     timestamp
> library(dplyr)
> #> 
> #> Attaching package: 'dplyr'
> #> The following objects are masked from 'package:stats':
> #> 
> #>     filter, lag
> #> The following objects are masked from 'package:base':
> #> 
> #>     intersect, setdiff, setequal, union
> url <- "https://ams3.digitaloceanspaces.com/uncomtrade/baci_hs92_1995.rds"
> rds <- "baci_hs92_1995.rds"
> if (!file.exists(rds)) try(download.file(url, rds))
> d <- readRDS("baci_hs92_1995.rds")
> rds_has_usa <- any(grepl("usa", unique(d$reporter_iso)))
> rds_has_usa
> #> [1] TRUE
> dir <- "parquet/baci_hs92"
> d %>% 
>   group_by(year, reporter_iso) %>% 
>   write_dataset(dir, hive_style = F)
> parquet_has_usa <- any(grepl("usa", list.files(paste0(dir, "/1995"))))
> parquet_has_usa
> #> [1] FALSE
> {code}
> _Created on 2021-06-24 by the [reprex package](https://reprex.tidyverse.org) (v2.0.0)_



--
This message was sent by Atlassian Jira
(v8.3.4#803005)