[jira] [Created] (ARROW-15743) [R] `skip` not connected up to `skip_rows` on open_dataset despite error messages indicating otherwise

["Nicola Crane (Jira)" <ji...@apache.org>]

Nicola Crane created ARROW-15743:
------------------------------------

             Summary: [R] `skip` not connected up to `skip_rows` on open_dataset despite error messages indicating otherwise
                 Key: ARROW-15743
                 URL: https://issues.apache.org/jira/browse/ARROW-15743
             Project: Apache Arrow
          Issue Type: Bug
          Components: R
            Reporter: Nicola Crane


If I open a dataset of CSVs with a schema, the error message tells me to supply {{`skip = 1`}} if my data contains a header row (to prevent it being read in as data), but only {{skip_rows = 1}} actually works.

{code:r}

library(arrow)
library(dplyr)

td <- tempfile()
dir.create(td)
write_dataset(mtcars, td, format = "csv")

schema <- schema(mpg = float64(), cyl = float64(), disp = float64(), hp = float64(), 
    drat = float64(), wt = float64(), qsec = float64(), vs = float64(), 
    am = float64(), gear = float64(), carb = float64())


open_dataset(td, format = "csv", schema = schema) %>%
  collect()
#> Error in `handle_csv_read_error()`:
#> ! Invalid: Could not open CSV input source '/tmp/RtmppZbpeF/file6cec135ed29c/part-0.csv': Invalid: In CSV column #0: Row #1: CSV conversion error to double: invalid value 'mpg'
#> /home/nic2/arrow/cpp/src/arrow/csv/converter.cc:550  decoder_.Decode(data, size, quoted, &value)
#> /home/nic2/arrow/cpp/src/arrow/csv/parser.h:123  status
#> /home/nic2/arrow/cpp/src/arrow/csv/converter.cc:554  parser.VisitColumn(col_index, visit)
#> /home/nic2/arrow/cpp/src/arrow/csv/reader.cc:463  arrow::internal::UnwrapOrRaise(maybe_decoded_arrays)
#> /home/nic2/arrow/cpp/src/arrow/compute/exec/exec_plan.cc:445  iterator_.Next()
#> /home/nic2/arrow/cpp/src/arrow/record_batch.cc:336  ReadNext(&batch)
#> /home/nic2/arrow/cpp/src/arrow/record_batch.cc:347  ReadAll(&batches)
#> ℹ If you have supplied a schema and your data contains a header row, you should supply the argument `skip = 1` to prevent the header being read in as data.

open_dataset(td, format = "csv", schema = schema, skip = 1) %>%
  collect()
#> Error: The following option is supported in "read_delim_arrow" functions but not yet supported here: "skip"

open_dataset(td, format = "csv", schema = schema, skip_rows = 1) %>%
  collect()
#> # A tibble: 32 × 11
#>      mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
#>    <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#>  1  21       6  160    110  3.9   2.62  16.5     0     1     4     4
#>  2  21       6  160    110  3.9   2.88  17.0     0     1     4     4
#>  3  22.8     4  108     93  3.85  2.32  18.6     1     1     4     1
#>  4  21.4     6  258    110  3.08  3.22  19.4     1     0     3     1
#>  5  18.7     8  360    175  3.15  3.44  17.0     0     0     3     2
#>  6  18.1     6  225    105  2.76  3.46  20.2     1     0     3     1
#>  7  14.3     8  360    245  3.21  3.57  15.8     0     0     3     4
#>  8  24.4     4  147.    62  3.69  3.19  20       1     0     4     2
#>  9  22.8     4  141.    95  3.92  3.15  22.9     1     0     4     2
#> 10  19.2     6  168.   123  3.92  3.44  18.3     1     0     4     4
#> # … with 22 more rows

{code}




--
This message was sent by Atlassian Jira
(v8.20.1#820001)