[jira] [Created] (ARROW-15124) default TZ parsing woes in CSV reader

["Carl Boettiger (Jira)" <ji...@apache.org>]

Carl Boettiger created ARROW-15124:
--------------------------------------

             Summary: default TZ parsing woes in CSV reader
                 Key: ARROW-15124
                 URL: https://issues.apache.org/jira/browse/ARROW-15124
             Project: Apache Arrow
          Issue Type: Bug
          Components: R
    Affects Versions: 6.0.1
            Reporter: Carl Boettiger


I am attempting to use open_dataset() on a large collection of CSV files in which a timestamp column sometimes has a date format and sometimes a timezone format.

readr is fine reading these both in with a col_type set to "timestamp" (i.e. see below), but arrow_read_csv insists the one must use tz="UTC" while the other must not use tz="UTC" in order for the schema to be valid.  Easiest to see this in a simple example:


{code:java}
x <- tempfile()
df <- data.frame(time = '2021-02-01T00:00:00Z')
readr::write_csv(df, x)
schema = arrow::schema(time = timestamp("s", ""))

# ERROR cannot parse w/o tz="UTC" in the schema:
arrow::read_csv_arrow(x,schema = schema, skip=1) 

df2 <- readr::read_csv(x, col_types="T")  # works fine{code}
{code:java}
df <- data.frame(time = '2021-02-01')
readr::write_csv(df, x)
## ERROR cannot parse w/ tz="UTC" :
schema = arrow::schema(time = timestamp("s", "UTC")) 
arrow::read_csv_arrow(x,schema = schema, skip=1)

## Once again, readr has no issues:
df2 <- readr::read_csv(x, col_types="T")
 {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)