[jira] [Commented] (ARROW-15879) [R] passing a schema calls open_dataset to fail on hive-partitioned csv files

["Sam Albers (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-15879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520706#comment-17520706 ] 

Sam Albers commented on ARROW-15879:
------------------------------------

I did some digging to the extent that I added a test that captured this failures here: [https://github.com/apache/arrow/pull/12831]

I can confirm that this does not happen when `format = 'parquet'`. The error message is coming from [here|https://github.com/apache/arrow/blob/master/cpp/src/arrow/dataset/file_csv.cc#L212] but that is about as far as I got. I think this is also related to ARROW-14743

> [R] passing a schema calls open_dataset to fail on hive-partitioned csv files
> -----------------------------------------------------------------------------
>
>                 Key: ARROW-15879
>                 URL: https://issues.apache.org/jira/browse/ARROW-15879
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: R
>    Affects Versions: 7.0.0, 7.0.1
>            Reporter: Carl Boettiger
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> Consider this reprex:
>  
> Create a dataset with hive partitions in csv format with write_dataset() (so cool!):
>  
> {code:java}
> library(arrow)
> library(dplyr)
> path <- fs::dir_create("tmp")
> mtcars %>% group_by(gear) %>% write_dataset(path, format="csv")## works fine, even with 'collect()'
> ds <- open_dataset(path, format="csv")## but pass a schema, and things fail
> df <- open_dataset(path, format="csv", schema = ds$schema, skip_rows=1)
> df %>% collect()
>  {code}
> In the first call to open_dataset, we don't pass a schema and things work as expected. 
> However, csv files often need a schema to be read in correctly, particularly with partitioned data where it is easy to 'guess' the wrong type.  Passing the schema though confuses open_dataset, because the grouping column (partition column) isn't found on the individual files even though it is mentioned in the schema!
> Nor can we just omit the grouping column from the schema, since then it is effectively lost from the data. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)